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I.  INTRODUCTION 


“Price  is  what  you  pay;  value  is  what  you  get.”  — Warren  Buffett 

The  Federal  Acquisition  Regulation  (FAR)  established  that  members  of  a  federal 
acquisition  team  are  responsible  for  utilizing  “personal  initiative  and  sound  business 
judgment  in  providing  the  best  value  product  or  service  to  meet  the  customer’s  needs” 
(FAR,  2014,  Part  1.102[d]).  What  constitutes  “best  value”  is  also  defined  by  the  FAR  in 
Part  2  as  “the  expected  outcome  of  an  acquisition  that,  in  the  government’s  estimation, 
provides  the  greatest  overall  benefit  in  response  to  the  requirement”  (FAR,  2014,  Part 
2.101).  What  goes  without  definition  in  the  FAR  is  a  clear  and  precise  means  for 
acquiring  best  value  for  the  requiring  party. 

The  current  Department  of  Defense  (DOD)  acquisition  environment  is  strained 
under  the  pressure  of  budget  restrictions,  limited  resources,  and  increasing  bureaucracy. 
The  result  is  an  overworked  and  undersupplied  workforce  that  is  still  trying  to  understand 
how  to  efficiently  and  effectively  garner  best  value  for  the  government.  Policy  and 
procedure  have  dominated  the  acquisition  cycle  in  efforts  to  produce  contracts  that 
deliver  acceptable  performance  with  timely  procurement  administrative  lead  times 
(PALT);  yet,  every  year,  the  Government  Accountability  Office  (GAO)  finds  continual 
evidence  of  underperforming  contracts  and  increasing  PALT.  In  this  research  we  explore 
how  best  value  is  theoretically  accomplished  using  contracting,  how  this  impacts  the 
current  state  of  the  DOD  contracting  workforce,  and  how  the  DOD  is  measuring  the 
outcomes  of  completed  contracts.  The  purpose  of  this  study  is  to  measure  if  the  use  of 
lowest  price  technically  acceptable  (LPTA)  or  tradeoff  methods  impact  the  quality  of 
contractor  performance  on  DOD  contracts  in  one  faction  of  the  United  States  (U.S.) 
Army. 
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A.  PROBLEM  STATEMENT 

Industry  and  government  have  differing  views  regarding  the  quality  of  LPTA  in 
evaluation  of  competitive  acquisitions.  The  government  views  LPTA  as  a  means  of 
utilizing  limited  resources  and  spending  only  the  necessary  taxpayer  dollars.  “Contractors 
complain  that  low-price/technically  acceptable  source  selection  results  in  inferior 
perfonnance  or  quality  because  contracts  are  awarded  to  the  most  cut-rate  bidders”  (Cox, 
2015,  para.  5).  Industry  is  showing  a  general  complaint  that  LPTA  results  in  lower- 
quality  delivery  on  contracts,  and  tradeoff  is  the  better  means  for  government  and 
industry  to  work  together. 

B.  PURPOSE 

This  study  aims  to  define  the  relationship  between  the  best  value  continuum 
(LPTA  and  tradeoff  methodologies)  and  the  outcome  of  competitively  awarded  contracts. 
By  having  a  better  understanding  of  these  methods  and  the  outcomes  of  the  contracts  they 
impact,  acquisition  teams  can  make  more  infonned  decisions  regarding  which 
methodology  to  employ  for  future  source  selections.  The  defense  budget  appears  to  only 
be  shrinking  in  the  foreseeable  future;  government  teams  could  benefit  from  verifying  if 
LPTA,  which  is  touted  as  saving  the  government  money  in  the  source  selection  process, 
actually  results  in  acceptable  contract  outcomes  or  if  tradeoff  gives  the  government  the 
best  value. 

This  study  aims  to  investigate  whether  or  not  the  type  of  methodology  for 
obtaining  best  value  procurement  has  a  measurable  impact  on  the  documented 
performance  or  outcome  deliverables  of  that  contract.  Given  the  limitations  and  gradually 
shrinking  defense  budgets  of  late,  this  study’s  results  could  benefit  future  acquisition 
teams  that  will  decide  which  methodology  to  employ  for  a  source  selection. 
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C.  RESEARCH  OBJECTIVES  AND  QUESTIONS 

The  objective  of  this  research  is  to  address  if  the  source  selection  methodology 
(either  LPTA  or  tradeoff)  has  an  effect  on  the  outcome  of  the  contract  (quality  of 
contractor  perfonnance)  measured  in  the  Contractor  Performance  Assessment  Reporting 
System  (CPARS).  The  following  research  questions  are  used  to  guide  this  endeavor: 

1 .  Does  the  utilization  of  LPTA  or  tradeoff  impact  the  outcome  of 
competitively  awarded  contracts? 

2.  Are  acquisition  teams  making  knowledgeable  decisions  regarding  which 
best  value  methodology  is  most  appropriate  for  their  source  selection? 

D.  METHODOLOGY 

The  scope  of  this  data  gathering  was  limited  to  contracts  awarded  for  programs 
under  the  Army  Program  Executive  Office  Enterprise  Information  Systems  (PEO  EIS) 
that  exceeded  the  simplified  acquisition  threshold  (SAT),  currently  set  at  $150,000  (FAR, 
2014,  Part  2.101).  Contracts  must  have  been  competitively  awarded,  and  utilized  either 
LPTA  or  tradeoff  methodologies  to  determine  award.  The  contracts  must  have  been 
closed  out  as  of  the  beginning  of  calendar  year  2015,  so  that  final  CPARS  reports  could 
be  obtained.  Contract  files  must  contain  pre-award  documentation,  source  selection  plans, 
contract  award  documents,  and  CPARS  reports,  at  a  minimum,  to  be  utilized  for  this 
study. 

The  first  effort  to  gather  an  appropriate  list  of  contracts  for  PEO  EIS  was  via  a 
query  from  the  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG),  for 
contracts  that  were  awarded  via  competitive  procurement  for  PEO  EIS  for  $150,000  or 
more,  for  which  entries  have  been  made  into  CPARS  or  other  post-award  performance 
evaluation.  From  that  list,  the  contracts  were  found  in  Electronic  Document  Access 
(EDA),  the  online  document  repository  for  awarded  contracts.  Electronic  documents  were 
acquired  from  the  databases  at  the  participating  Anny  Contracting  Command  (ACC)- 
Rock  Island  (ACC-RI)  office  at  Rock  Island  Arsenal  in  Illinois,  and  the  ACC-New  Jersey 
(ACC-NJ)  office.  The  contract  files  were  also  requested  from  the  requiring  program 
offices  for  the  contracts,  to  include  documents  such  as  the  Acquisition  Plans/Acquisition 
Strategies,  Source  Selection  Plans,  Sections  L  (instructions,  conditions,  and  notices  to 
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offerors  or  respondents)  and  M  (evaluation  factors  for  award)  of  the  contracts,  Pre-and- 
Post  Negotiation  Memorandums,  and  other  pre-award  documentation.  A  memorandum 
from  the  current  Program  Executive  Office  for  the  Army  PEO  Enterprise  Infonnation 
Systems  was  provided  to  participating  parties,  confirming  his  approval  and  the  non- 
attributional  nature  of  this  study. 

E.  BENEFITS  AND  LIMITATIONS 

The  DOD  is  striving  to  demonstrate  good  business  judgment  and  move  toward 
smart  execution  of  funds — to  hold  more  power  in  defense  buying.  The  latest  publication 
in  the  series  of  Better  Buying  Power  was  released  on  April  9,  2015.  Within  this  release  of 
Better  Buying  Power  3.0  from  the  Undersecretary  of  Defense  for  Acquisition, 
Technology  &  Logistics  (USD  (AT&L))  Frank  Kendall,  under  the  “Incentivize 
Innovation  in  Industry  and  Government”  (Kendall,  2015c,  p.  2)  initiative,  there  is  a  goal 
to  “provide  clear  ‘best  value’  definitions  to  industry”  (Kendall,  2015c,  p.  18).  This 
guidance  is  working  to  reform  the  DOD  to  put  best  value  at  the  forefront  of  acquisition 
concerns.  While  results  from  these  reforms  will  not  be  seen  for  years  to  come,  the  efforts 
being  put  forth  are  indicative  of  a  massive  priority  shift  for  DOD  procurement. 
Understanding  how  best  value  contracting  decisions  impact  the  outcomes  of  the  overall 
contract  would  be  beneficial  to  future  acquisition  teams  seeking  efficient  means  of 
achieving  best  value. 

The  results  from  the  last  BBP  2.0  included  a  best  value  process  manual  that  is 
scheduled  to  be  added  to  the  official  Source  Selection  Guide,  which  will  hopefully  result 
in  reduced  fear  of  utilizing  tradeoff  methods  and  negotiations  for  appropriate 
procurements.  With  clearer  indication  for  the  outcomes  of  a  contract,  there  is  a  chance  for 
be  reduced  risk  of  contract  waste,  fraud  and  abuse,  and  bigger  potential  for  achieving  best 
value  for  the  government. 

While  the  goal  of  this  research  is  to  define  the  relationship  between  elements  of 
the  best  value  continuum  and  the  outcomes  of  competitively  awarded  contracts,  there  are 
limitations  to  the  contract  data  being  considered.  This  study  is  only  considering  Army 
contracts,  for  the  PEO  EIS  (which  is  discussed  in  Chapter  III),  from  select  Army 
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contracting  commands.  This  limitation  leaves  a  large  amount  of  data  not  collected  nor 
considered  for  this  study.  The  specific  contract  data  reviewed  is  limited  to  contracts  that 
exceed  the  SAT,  were  perfonned  between  2001  and  2014  (completely)  and  also  have 
complete  and  accessible  CPARS  reports.  This  limitation  leaves  contracts  with  period  of 
perfonnance  dates  prior  to  2001,  contracts  below  the  SAT  or  contracts  with  informal  past 
perfonnance  information  not  gathered  or  analyzed. 

F.  SUMMARY 

As  Warren  Buffett  stated  in  his  2008  Berkshire  Hathaway,  Inc.  Annual  Report 
and  Letter  to  the  Shareholders,  “Price  is  what  you  pay;  value  is  what  you  get.”  Best  value 
is  clearly  defined  by  the  FAR,  as  are  the  categories  on  the  best  value  Continuum:  LPTA 
and  tradeoff.  What  remains  to  be  measured  is  whether  LPTA  or  tradeoff  will  result  in 
different  levels  of  value  for  the  government.  Buffett  continues  in  his  letter  by  saying, 
“Whether  we’re  talking  about  socks  or  stocks,  I  like  buying  quality  merchandise  when  it 
is  marked  down”  (Buffett,  2008,  p.  5).  The  same  mantra  can  be  applied  to  government 
acquisition  in  that  acquisition  teams  must  be  stewards  of  the  taxpayer  dollar  and  aim  to 
receive  the  best  value  for  the  dollar  being  spent.  The  aim  of  this  study  is  to  determine  if 
one  method  is  better  than  the  other  in  endeavoring  to  be  conscientious  wardens  of  the 
American  dollar. 

This  chapter  discussed  the  background  information,  the  problem  statement,  the 
purpose  of  this  research,  the  methodology,  and  the  benefits  and  limitations  of  this  study. 
The  next  chapter  contains  the  literature  review  for  the  basic  concepts  and  issues  relevant 
to  this  project,  including  source  selection  methodologies,  the  modern  acquisition 
workforce,  how  government  is  working  to  reform  best  value  contracting,  as  well  as 
criticisms  from  industry  regarding  best  value  contracting.  The  importance  of  past 
perfonnance  infonnation  is  discussed,  and  its  impact  on  source  selection  evaluation 
boards. 
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II.  LITERATURE  REVIEW 


The  purpose  of  this  chapter  is  to  highlight  key  information  that  academia  has 
already  established  to  be  true  or  of  utmost  important  to  the  field  of  best  value  contracting. 
It  begins  with  a  review  of  the  source  selection  methodologies  utilized  in  best  value 
contracting,  LPTA  and  tradeoff,  and  the  factors  that  indicate  proper  utilization  of  each 
methodology.  Additionally,  this  chapter  provides  a  discussion  on  the  modern  acquisition 
workforce  and  its  level  of  comfort  with  utilizing  best  value  procurements,  and  the  need 
for  acquisition  education.  This  chapter  explores  the  DOD’s  efforts  to  reform  best  value 
contracting,  as  well  as  industry’s  criticisms  of  DOD  contract  management  processes. 
Finally  the  chapter  will  conclude  with  a  discussion  on  the  importance  of  past 
perfonnance  information. 

A.  SOURCE  SELECTION  PREPARATION 

The  contract  management  process  is  composed  of  six  basic  phases:  procurement 
planning,  solicitation  planning,  solicitation,  source  selection,  contract  administration,  and 
contract  closeout.  When  an  acquisition  team  fonns  and  begins  the  contract  management 
process,  the  team  follows  the  standard  six  phases  in  order,  beginning  with  procurement 
planning.  The  team  will  detennine  what  is  being  procured,  the  details  of  the  procurement, 
and  “which  business  needs  can  be  best  met  by  procuring  products  or  services  outside  the 
organization”  (Rendon  &  Garrett,  2005,  p.  5).  Upon  entering  the  solicitation  planning 
phase,  the  team  begins  “the  process  of  preparing  the  documents  needed  for  source 
selection”  (p.  5)  and  starts  to  make  crucial  decisions  regarding  how  the  evaluation  will 
commence  during  the  source  selection  phase.  The  team  will  need  to  determine  not  only 
the  evaluation  factors  but  also  the  relative  importance  of  those  evaluation  factors.  The 
team  will  then  detennine  whether  to  move  forward  with  the  LPTA  or  tradeoff 
methodologies,  depending  on  a  variety  of  factors  pertaining  to  the  acquisition.  The  source 
selection  methodology  will  be  identified  in  section  M  of  the  contract  that  is  released  with 
the  Request  for  Proposal  (RFP)  to  industry  during  the  solicitation  phase. 
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Section  15  of  the  FAR,  Contracting  by  Negotiation,  discusses  source  selection 
procedures  and  how  to  detennine  the  evaluation  factors  pertinent  to  a  particular 
acquisition.  Specifically,  section  15.101  outlines  the  suitable  situations  for  both  the 
tradeoff  and  LPT  A  source  selection  processes.  The  purpose  of  section  15.101  is  to  assist 
agencies  in  obtaining  best  value  by  identifying  the  varying  methods  that  can  be  utilized  to 
that  end. 

This  project  focuses  primarily  on  the  decisions  made  before  and  during  the  source 
selection  phase,  guided  by  section  15,  and  how  those  decisions  impact  the  contract 
outcomes.  Before  moving  forward  into  the  source  selection  phase,  the  acquisition  team 
must  determine  if  the  acquisition  should  be  evaluated  using  the  LPTA  or  tradeoff  source 
selection  methodologies.  In  order  to  make  that  decision,  the  team  must  be  familiar  with 
the  methodologies  and  how  to  detennine  if  an  acquisition  is  appropriate  for  either. 

B.  THE  BEST  VALUE  CONTINUUM 

Outlined  in  FAR  Part  15.101,  “best  value”  refers  to  the  continuum  for  proposal 
evaluation  on  which  there  are  two  general  types  of  source  selection  methods:  tradeoff  and 
LPTA.  To  find  the  appropriate  place  on  this  continuum  for  an  acquisition,  multiple 
factors  and  their  relative  importance  must  be  defined,  to  include  the  cost  or  price,  the 
level  of  requirements  definition,  the  level  of  risk,  and  how  dominant  the  roles  of  non-cost 
factors  will  be  played  out  in  the  evaluation.  A  2010  GAO  report  revealed  that  the  DOD 
chose  a  best  value  process  for  approximately  “95%  of  its  new,  competitively  awarded 
contracts  on  which  it  had  obligated  $25M  or  more  in  fiscal  year  2009”  (GAO,  2010,  p.  7). 
The  remaining  5%  were  completed  using  Sealed  Bidding,  which  is  a  competitive  process 
in  which  award  is  made  to  a  bidder  whose  bid  conforms  to  the  invitation  for  bid  and  is 
most  advantageous  to  the  government,  considering  only  price  and  price-related  factors 
included  in  the  solicitation  (GAO,  2010,  p.  7).  Therefore  the  DOD  is  extensively  utilizing 
best  value  source  selection  methodologies  found  on  the  best  value  continuum. 

The  best  value  continuum  is  a  spectrum  that  is  based  on  the  importance  of  price 
and  non-cost  factors,  relative  to  price;  as  the  importance  of  price  decreases,  the  tradeoff 
process  is  more  strongly  implemented.  LPTA  and  tradeoff  reside  at  different  positions  on 
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the  continuum,  based  on  the  varying  importance  of  non-cost  factors.  If  the  sum  of  non¬ 
cost  factors  is  equal  to  or  more  important  than  price,  then  tradeoff  process  is  in  play, 
while  LPT  A  is  utilized  when  the  price  is  more  important  than  all  other  non-cost  factors 
combined  (GAO,  2014b,  p.  4).  The  best  value  continuum  is  defined  clearly  at  FAR 
15.101, 


An  agency  can  obtain  best  value  in  negotiated  acquisitions  by  using  any 
one  or  a  combination  of  source  selection  approaches.  In  different  types  of 
acquisitions,  the  relative  importance  of  cost  or  price  may  vary.  For 
example,  in  acquisitions  where  the  requirement  is  clearly  definable  and  the 
risk  of  unsuccessful  contract  performance  is  minimal,  cost  or  price  may 
play  a  dominant  role  in  source  selection.  The  less  definitive  the 
requirement,  the  more  development  work  required,  or  the  greater  the 
performance  risk,  the  more  technical  or  past  performance  considerations 
may  play  a  dominant  role  in  source  selection.  (FAR,  2014,  Part  15.101) 

The  Best  Value  Continuum,  seen  in  Figure  1,  is  a  spectrum  with  points  located 
based  on  the  importance  of  price  and  non-cost  factors: 


Figure  1.  Source  Selection  Processes  on  the  Best  Value  Continuum  (from 

GAO,  2014b) 


According  to  the  design  of  the  best  value  continuum,  as  the  importance  of  price 
decreases  the  non-cost  factors  becoming  increasingly  more  important,  thereby  indicating 
the  appropriate  environment  for  a  tradeoff  procurement.  However,  as  the  procurement’s 
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importance  of  cost  grows  and  outweighs  the  importance  of  non-cost  factors,  LPTA 
becomes  the  more  appropriate  choice  for  the  procurement. 

1.  Lowest  Price  Technically  Acceptable 

LPTA  is  a  best  value  source  selection  method  utilized  when  “the  best  value  is 
expected  to  result  from  selection  of  the  technically  acceptable  proposal  with  the  lowest 
evaluated  price”  (FAR,  2014,  Part  15.101-2).  Requirements  are  typically  well  defined, 
the  technical  requirements  are  clear  and  there  is  no  need  for  tradeoffs. 

Fewer  resources  (time,  money,  and  people)  are  necessary  to  perform  a  LPTA 
evaluation.  Relatively  inexperienced  acquisition  personnel  can  perform  the  evaluation. 
Under  a  LPTA  requirement,  contractual  expectations  and  technical  requirements  are  not 
easy  to  misunderstand,  and  therefore  it  is  easy  to  determine  if  a  proposal  is  technically 
acceptable.  “LPTA  should  be  used  in  situations  where  DOD  would  not  realize  any  value 
from  a  proposal  exceeding  its  minimum  technical  or  perfonnance  requirements  and  that 
another  process  should  be  used  when  standards  or  performance  and  quality  are 
subjective”  (GAO,  2014b,  p.  6).  LPTA  is  frequently  used  for  purchase  of  commercial  off- 
the-shelf  (COTS)  products  and  services.  If  there  is  no  additional  benefit  from  paying 
more,  then  the  lowest  price  technically  acceptable  method  is  best. 

2.  Tradeoff 

Tradeoff  is  a  best  value  source  selection  evaluation  method  utilized  when  “it  may 
be  in  the  best  interest  of  the  Government  to  consider  award  to  other  than  the  lowest 
priced  offeror  or  other  than  the  highest  technically  rated  offeror”  (FAR,  2014,  Part 
15.101-1).  Requirements  are  typically  not  as  well  defined,  the  work  is  experimental  or  the 
government  is  looking  for  the  contractor  to  be  innovative  in  its  solutions.  The  solicitation 
for  a  tradeoff  acquisition  “shall  state  whether  all  evaluation  factors  other  than  cost  or 
price,  when  combined,  are  significantly  more  important  than,  approximately  equal  to,  or 
significantly  less  important  than  cost  or  price”  (FAR,  2014,  Part  15.101-1  [b],  [2]). 

Non-cost  factors  are  evaluation  factors  that  are  not  related  in  any  direct  way  to  the 
cost  of  a  contract.  Examples  include  past  perfonnance,  evaluation  of  the  technical 
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capability,  small  business  participation,  and  delivery  schedule.  Non-cost  factors  open  the 
door  for  discussions  between  the  government  and  the  offerors,  especially  in  negotiations, 
making  for  better  understanding  between  all  parties  of  the  requirements,  the  level  of 
expectations,  incentives,  and  other  pertinent  elements  of  a  successful  business 
relationship.  According  to  GAO  Report  11-8,  the  most  utilized  evaluation  factors  under 
tradeoff  were  found  to  be  Past  Performance,  Technical  Evaluation,  Timing  of  Delivery 
(where  delivery  of  a  final  product  or  service  was  crucial  to  the  contract’s  success), 
Reliability  of  perfonnance  in  hazard  zones,  and  Innovation.  Non-cost  factors  used  most 
often  were  Past  Performance  and  Technical  Capability  (GAO,  2010,  p.  13). 

Under  tradeoff,  there  is  need  for  better  communication  between  government  and 
contractor  regarding  requirements.  Better  discussion  surrounding  clarity  of  requirements 
and  chances  for  incentivizing  for  quality  contract  performance  and  outcomes  are  key  to  a 
successful  tradeoff.  Tradeoff  can  be  particularly  rewarding  for  requirements  that  are  not 
as  clear,  the  work  is  relatively  new,  or  the  government  is  looking  for  innovative 
approaches  from  industry.  Kendall  identifies  in  Better  Buying  Power  3.0  that  “whenever 
the  Warfighter  is  willing  to  pay  more  for  above  threshold  requirements  or  performance 
standards  and  may  benefit  from  an  innovative  and  technologically  superior  solution  to 
meet  their  mission  needs,  a  tradeoff  source  selection  process  between  cost  or  price  and 
non-cost  factors  is  optimal”  (Kendall,  2015c,  p.  2).  The  situations  that  afford  the  greatest 
chance  for  a  successful  tradeoff  are  clear,  but  too  often  overlooked  in  the  rush  to  make 
award  or  the  ever-increasing  pressure  of  budgetary  constraints.  Tradeoff  source  selection 
requires  more  time,  money  and  people  than  a  LPTA  source  selection.  The  tradeoff 
methodology  can  be  a  wonderful  means  of  achieving  innovative  solutions  that  lead  to 
technological  advances  for  the  DOD,  if  the  acquisition  teams  are  dedicated  to  the  chance 
of  achieving  a  quality  tradeoff  discussion.  In  order  to  achieve  the  desired  results  from 
discussions  regarding  non-cost  factors,  the  acquisition  team  must  understand  how  to 
properly  utilize  non-cost  factors  in  the  hopes  of  obtaining  best  value  for  the  government. 
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C.  SIGNIFICANCE  OF  NON-COST  FACTORS  AND  PROPER 

UTILIZATION 

The  easier  way  to  determine  savings  and  a  “better  deal”  is  when  the  price  can  be 
compared  between  two  like  items.  When  dealing  with  non-cost  factors,  that  simplicity  is 
lost  and  the  acquisition  team  must  determine  the  importance  of  factors  that  do  not  have 
an  objective  dollar  value,  and  furthermore,  determine  the  weight  of  those  non-cost  factors 
in  relation  to  the  dollar  value.  Thoroughly  identifying  the  importance  of  those  factors,  as 
well  as  understanding  the  reliability  of  that  data,  can  pay  off  handsomely  throughout  the 
evaluation  process. 

Evaluation  of  service  contracts  requires  a  different  level  of  quality  review — unlike 
a  product  contract  that  delivers  an  end  item  of  supply,  service  contracts  require  constant 
surveillance  and  monitoring  in  order  to  keep  minor  problems  from  escalating  beyond 
control  and  ultimately  resulting  in  loss  of  quality.  Past  performance  is  one  of  the  most 
utilized  non-cost  factors  for  evaluation,  which  can  call  into  question  the  reliability  of  that 
past  performance  information.  The  GAO  reported  in  the  09-374  report  that  past 
perfonnance  information  was  reluctantly  relied  upon  by  officials  “due,  in  part,  to  their 
skepticism  about  the  reliability  of  the  infonnation  and  difficulty  assessing  relevance  to 
specific  acquisitions”  (GAO,  2009,  p.  2).  If  government  was  not  properly  and 
consistently  reporting  on  the  perfonnance  of  the  contractors,  then  how  could  future 
source  selection  boards  benefit  from  the  knowledge  gleaned  from  that  performance?  Only 
by  standardizing  and  regulating  how  contractor  past  performance  is  reported  can  that  data 
be  relied  upon  to  provide  value  during  a  source  selection. 

Converting  non-cost  factors  to  a  dollar  value  is  a  concept  discussed  by  Kendall,  as 
we  will  explore  later  in  this  paper.  By  monetizing  performance  metrics,  there  is  increased 
customer  involvement  in  understanding  requirement  pricing,  thereby  giving  the  customer 
an  insight  into  how  the  evaluators  will  make  their  decision.  By  knowing  how  much  the 
customer  is  willing  to  pay,  based  on  the  value  of  the  increased  performance,  a  stronger 
justification  can  be  made  for  the  decisions  based  on  non-cost  factors,  to  potentially 
reduce  the  chance  for  successful  protests.  Further,  as  industry  is  better  aware  of  how 
decisions  are  made  regarding  non-cost  factors  and  what  is  most  important  to  the 
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government,  the  quality  of  proposals  could  increase  as  well  as  reduced  costs  for  the 
contractor  as  time  is  better  spent  on  proposal  development. 

Risk  is  a  non-cost  factor  that  may  not  be  typically  seriously  considered  for 
contracts  not  in  a  contingency-type  environment.  When  there  is  concern  that  a  contractor 
will  need  to  perfonn  on  time  and  in  a  dangerous  area,  risk  is  on  the  forefront  of 
everyone’s  mind.  Further  from  the  front  burner  is  the  concept  of  cost  risk,  which  is 
“meant  to  be  a  measure  of  the  degree  of  difference  between  an  offeror’s  proposed  cost 
and  the  government’s  developed,  most  probable  cost”  (Graham,  2007,  p.  32).  As 
proposals  are  received  and  the  gap  between  the  government’s  expectations  and  the 
contractor’s  proposal  are  measured,  cost  risk  can  be  identified  and  considered  as  a  non¬ 
cost  factor  during  discussions.  The  unreliability  of  a  government  estimate,  however, 
increases  the  risk  that  the  contract  will  not  be  fairly  priced  in  accordance  with  the 
contract  requirements.  As  non-cost  factors  are  understood  and  utilized  effectively, 
tradeoff  is  a  less  terrifying  process  than  its  reputation  touts. 

As  the  importance  of  non-cost  factors  increases  and  becomes  of  equal  to  or  more 
important  than  the  cost  factor,  the  source  selection  process  to  obtain  best  value  for  the 
government  becomes  a  tradeoff  process.  In  order  to  understand  the  importance  of  non¬ 
cost  factors,  the  acquisition  team  must  understand  the  degree  to  which  requirements  are 
defined,  the  limitations  of  budget,  the  degree  to  which  the  government  is  willing  to  flex 
on  each  factor — multiple  important  issues  need  to  be  discussed  by  the  acquisition  team, 
including  expectations  for  contractor  proposals.  The  Better  Buying  Power  (BBP) 
initiative  was  initially  released  by  the  USD,  Dr.  Ashton  Carter,  in  2010  (at  that  time,  the 
USD  (AT&L)).  BBP  identifies  seven  major  areas  that  USD  (AT&L)  is  pursuing  to 
“enhance  and  incentivize  efficiency”  (Carter,  2010,  p.  3),  thereby  inviting  industry  to 
work  together  with  government  in  the  tradeoff  process.  Government  should  be  making 
strides  towards  better  understanding  where  to  accommodate  “paying  more  (up  to  some 
amount)  than  the  minimum  price  bid  in  return  for  a  product  that  provides  more  than  the 
minimum  needed  performance”  (GAO,  2014b,  p.  6).  Acquisition  teams  may  or  may  not 
be  dedicating  ample  time  to  discussing  the  importance  of  the  factors  being  evaluated  in 
making  the  source  selection  decision,  or  are  not  thoroughly  analyzing  the  customer’s 
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needs  over  the  wants.  If  innovation  is  needed,  for  example,  then  price  may  not  the  most 
important  factor.  These  discussions  and  concerns  are  difficult  to  see  to  fruition  when  the 
issue  of  funding  is  looming  overhead. 

Policymakers  often  insist  that  the  best  way  to  define  “best  value”  is  in  monetary 
terms.  The  customer  and  the  acquisition  team  need  to  firmly  measure  the  extent  to  which 
more  money  earns  more  performance,  and  the  limit  to  where  dollars  provide  a  return  on 
investment.  While  the  requiring  party  is  the  one  benefitting  from  the  final  contract 
perfonnance,  the  source  selection  authority  is  the  party  making  the  decision  to  award  to 
the  offeror,  and  has  to  justify  that  decision.  Kendall  discussed  in  a  recent  Defense  AT&L 
Magazine  article,  “Getting  ‘Best  Value’  for  the  Warfighter  and  the  Taxpayer,”  the  protest 
fear  that  overcomes  acquisition  teams,  and  the  resulting  reliance  on  choosing  the  low-cost 
offer  purely  because  it  is  easy  to  defend:  “The  likely  bias  for  an  acquisition  official 
making  the  source  selection  is  to  take  the  lowest-price  offer;  it’s  much  easier  to  defend 
than  the  subjective  judgment  that  the  higher-cost  offeror  was  worth  the  difference  in 
price”  (Kendall,  2015b,  p.  2).  There  are  benefits  to  monetizing  performance  metrics, 
including  increased  customer  involvement  in  understanding  requirement  pricing  (“it 
forces  our  customers — the  operators  who  set  requirements — to  consider  how  much  they 
are  willing  to  pay  for  higher  performance”  [p.  2])  and  inviting  industry  to  be  intimately 
aware  of  the  importance  of  clear  pricing: 

We  want  industry  to  be  in  a  position  to  make  informed  judgments  about 
what  level  of  performance  to  offer.  The  easiest  way  to  accomplish  this  is 
to  tell  industry  exactly,  in  dollars  and  cents,  what  higher  levels  of 
perfonnance  are  worth  to  us.. ..Value  or  worth  to  the  buyer  has  nothing  to 
do  with  cost;  it  is  only  about  what  we  would  be  willing  to  pay  for 
something.  (Kendall,  2015b,  pp.  2-3) 

Beyond  the  open  communication  benefit  with  the  customer  and  with  industry, 
there  is  benefit  in  reduced  protests,  since  factors  previously  non-monetized  no  longer 
invite  criticism  on  basis  of  subjectivity.  “Avoiding  successful  protests  is  about  setting 
down  the  rules  for  source  selection,  following  them  religiously,  documenting  the 
decisions  we  make  so  we  can  explain  them  if  challenged,  and  maintaining  the  process 
integrity”  (Kendall,  2015b,  p.  2).  The  training  for  source  selection  teams  prior  to 


14 


performing  evaluations  only  revolves  around  the  ethical  issues  of  fairness.  The  pressure 
to  maintain  strict  ethical  conduct  and  avoid  potential  for  protest  may  inadvertently  lead 
acquisition  teams  to  take  an  “easier”  path  and  therefore  only  utilize  the  LPTA  method, 
even  if  the  requirements  are  not  appropriate  for  that  method  or  the  potential  for  gamering 
better  value  lie  in  utilizing  the  tradeoff  method. 

The  differences  in  using  LPTA  or  tradeoff  again  lie  in  the  solidity  of  the 
requirements  and  the  degree  of  flexibility  between  those  requirements  and  price. 
“Sometimes  LPTA  makes  sense  but  it  doesn’t  make  sense  if  we  are  willing,  as  we  usually 
are,  to  pay  a  little  more  for  a  much  better  product”  (Kendall,  2015b,  p.  3).  If  all  parties 
understand  what  the  government  is  looking  to  buy,  and  the  elements  on  which  they  are 
willing  to  be  flexible  and  pay  more,  there  is  greater  likelihood  that  the  finalized  contract 
will  be  most  appropriate  and  be  of  best  value  to  the  government  even  if  the  price  is 
slightly  higher.  Understanding  the  factors  involved  and  whether  there  is  a  necessity  for 
tradeoff  is  most  important,  as  “LPTA  is  appropriate  when  we  have  well-defined  standards 
of  perfonnance  and  we  do  not  place  any  value  on,  and  are  therefore  unwilling  to  pay  for, 
higher  performance”  (p.  3).  If  there  is  potential  for  significant  performance  benefit  by 
paying  just  a  little  more,  most  acquisition  professionals  with  good  business  judgment 
would  agree  that  the  taxpayer  and  customer  are  amenable  to  that  tradeoff. 

The  DOD  is  clearly  making  significant  efforts  to  improve  upon  best  value 
contracting.  Even  with  these  increased  efforts  to  properly  remind  the  acquisition 
community  of  the  proper  utilization  of  both  best  value  methods,  however,  use  of  LPTA  is 
on  the  rise  as  of  recent. 

D.  INCREASING  USE  OF  LPTA 

Year  after  year,  the  DOD  is  stricken  with  mandates  to  reduce  costs  and  reduce 
funding  for  programs,  which  leave  those  programs  to  make  do  with  less  funding  for  a 
mission  that  is  equal  to  or  greater  than  the  year  before.  The  changing  technological 
landscape  and  changing  missions  needs  leave  acquisition  professionals  with  few  options 
but  to  further  employ  use  of  LPTA.  “DOD  officials  acknowledged  several  challenges  in 
using  the  best  value  tradeoff  process  such  as  the  difficulties  in  developing  meaningful 
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evaluation  factors,  the  additional  time  investment  needed  to  conduct  best  value 
procurements,  and  the  business  judgment  required  of  acquisition  staff  when  compared  to 
other  acquisition  approaches”  (GAO,  2010,  p.  21).  There  is  therefore  a  belief  amongst 
DOD  officials  that  LPTA  acquisitions  require  fewer  resources  to  implement,  while 
tradeoff  typically  requires  more  time  to  evaluate  factors,  weigh  significance  of  non-cost 
factors,  and  engage  in  meaningful  negotiations — time  and  resources  which  the 
government  cannot  afford  for  multiple  simultaneous  procurements. 

LPTA  has  been  gradually  increasing,  used  at  nearly  the  same  rate  as  tradeoff, 
suggesting  that  the  simplified  method  is  becoming  more  accessible  for  teams  that  are 
seeking  efficient  evaluations  to  reach  award  to  vendors  faster  with  fewer  complex 
elements.  “In  making  tradeoff  decisions,  GAO  found  that  DOD  selected  a  lower  priced 
proposal  nearly  as  often  as  it  selected  a  higher  technically  rated,  but  more  costly 
proposal”  (GAO,  2010,  p.  2).  Kendall  defines  in  his  recent  memorandum,  Appropriate 
Use  of  Lowest  Priced  Technically  Acceptable  Source  Selection  Process  and  Associated 
Contract  Type,  that  LPTA  is  “the  most  appropriate  source  selection  process  to  apply  only 
when  there  are  well-defined  requirements,  the  risk  of  unsuccessful  contract  perfonnance 
is  minimal,  price  is  a  significant  factor  in  the  source  selection,  and  there  is  neither  value, 
need,  nor  willingness  to  pay  for  higher  performance”  (Kendall,  2015a,  p.  1).  Complex, 
poorly-defined  requirements  where  the  DOD  is  seeking  innovative  solutions  therefore 
does  not  meet  the  requirements  for  use  of  LPTA.  With  increasing  pressure  to  award 
contracts  quickly  and  efficiently,  however,  LPTA  would  understandably  be  utilized  more 
frequently  than  a  time-consuming  negotiation  method,  and  can  be  tempting  to  use  always, 
even  if  the  procurement  is  not  appropriate  for  that  source  selection  method. 

The  capability  and  competence  of  the  modem  acquisition  workforce  is  essential  to 
developing  successful  contracts.  The  acquisition  team,  comprising  contracting  and  non¬ 
contracting  personnel,  is  charged  with  selecting  the  most  appropriate  source  selection 
methodology  in  order  to  gain  best  value  for  the  government.  With  this  known 
responsibility,  how  certain  can  we  be  that  any  given  acquisition  team  is  capable  of 
selecting  the  most  appropriate  methodology? 
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E.  CURRENT  ACQUISITION  PERSONNEL:  EXPERIENCE, 

KNOWLEDGE,  AND  COMFORT  WITH  BEST  VALUE 

PROCUREMENTS 

The  DOD  in  201 1  “was  put  on  the  verge  of  a  retirement-driven  talent  drain  in  this 
workforce  after  11  consecutive  years  of  downsizing”  (GAO,  2012,  p.  2).  Following  the 
massive  reduction  of  the  workforce  and  its  assets,  the  U.S.  found  itself  engaging  in 
multiple  wars  worldwide  and  a  technological  boom  that  required  excessive  growth  and 
purchasing  in  order  to  remain  a  leader  in  worldwide  defense. 

An  underlying  issue  of  the  acquisition  workforce  is  the  lack  of  confidence  in 
executing  complex  contracting  actions.  The  DOD  reacted  to  the  downsizing  of  personnel 
in  the  1990s  by  increasing  its  acquisition  workforce  positions  from  133,102  in  fiscal  year 
(FY)09  to  151,355  in  FY13,  just  in  the  contracting  career  field  (GAO,  2014b,  p.  1 7).  This 
massive  hiring  of  new  contracting  personnel  could  pay  dividends  in  the  future,  but  the 
current  landscape  shows  a  general  discomfort  with  executing  complex  contracting 
actions,  as  there  has  not  been  enough  time  (nor  resources  available)  to  get  these 
unseasoned  contracting  personnel  up  to  speed. 

LPTA  is  typically  a  more  comfortable  process,  since  it  takes  less  intimate 
knowledge  of  a  product  or  service  requirement  and  rather  relies  more  heavily  on  the  cost 
evaluation.  A  general  discomfort  with  intimate  knowledge  of  a  requirement,  in  addition 
to  limited  time  and  resources  to  work  on  a  single  procurement  when  hundreds  of  other 
actions  are  standing  by,  leads  acquisition  teams  to  utilize  LPTA  over  tradeoff. 

Diversity  is  not  the  issue  in  the  workforce,  but  rather  a  wide  gap  in  the  levels  of 
experience  in  each  group;  while  a  majority  of  the  workforce  is  currently  near  or  eligible 
for  retirement,  the  remaining  personnel  do  not  have  the  on-the-job  experience  necessary 
to  facilitate  complex  procurements  confidently,  which  has  resulted  in  an  increase  of 
LPTA  procurements  for  their  simplicity  as  well  as  a  number  of  best  value  procurements 
being  overall  incorrectly  utilized. 

According  to  GAO  report  14-584,  “Several  contracting  and  program  officials  said 
that  their  commands  gave  more  attention  to  whether  LPTA  is  an  alternative  option  in 
light  of  declining  budgets  and  efficiency  initiatives”  (GAO,  2014b,  p.  1).  GAO  found  that 
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for  contracts  valuing  $25M  or  more  there  is  an  increased  use  of  LPTA  between  2009  and 
2013  (increase  by  10%)  and  a  decrease  in  use  of  tradeoff  (decrease  by  11%)  (p.  9). 
Several  agencies,  including  the  Army,  Navy  and  Air  Force  (p.  13)  cite  the  Better  Buying 
Power  initiative  in  the  use  of  LPTA  as  fewer  personnel  resources  are  required  for  a  LPTA 
evaluation  than  a  tradeoff,  and  even  less-experienced  acquisition  personnel  can  perform  a 
LPTA  evaluation — a  solid  example  of  “doing  more  without  more”  (p.  5).  DOD  officials 
believe  that  LPTA  requires  less  manpower  to  review  complex  non-cost  evaluation 
factors,  less  time  to  review  proposals,  less  time  to  make  award.  In  an  environment  where 
there  is  less  of  everything — time,  money  and  manpower — and  an  increasingly  larger 
mission  with  needs  to  be  met,  the  DOD  has  invested  less  in  training  its  workforce  to 
handle  challenging  acquisitions  and  rather  to  employ  LPTA  whenever  possible.  “DOD 
predominantly  used  best  value  processes  in  FY13,  but  increased  its  use  of  LPTA  for 
higher  dollar  contracts  since  FY09”  (GAO,  2014b,  p.  9)  for  obligations  of  $25M  or  more, 
indicating  that  even  though  best  value  is  still  the  main  goal,  acquisition  personnel  are 
more  often  choosing  the  road  more  easily  traveled.  Even  though  the  DOD  budget  is  being 
reduced  year  after  year,  the  mission  remains  the  same;  therefore,  to  do  “more  without 
more”  there  is  a  greater  need  for  intelligent  business  decisions  and  strong  negotiating 
skills  to  get  bigger  returns  on  the  American  taxpayer  dollar. 

Quality  of  education  of  the  acquisition  workforce  has  taken  a  back  seat,  given  the 
rush  to  get  contracts  awarded  combined  with  the  influx  of  new,  unseasoned  acquisition 
personnel.  The  range  of  educational  courses  offered  by  the  Defense  Acquisition 
University  (DAU)  for  source  selection  are  mandatory  for  contracting  personnel,  but  offer 
only  what  the  classroom  can — theoretical  application.  According  to  officials  from  the 
Assistance  Secretary  for  the  Army  for  Acquisition,  Logistics  &  Technology  (ASA 
(ALT)),  “on-the-job  training  provides  important  exposure  for  less  experienced  acquisition 
staff  to  the  source  selection  decision  making  processes”  (GAO,  2014b,  p.  21).  So  while 
contracting  personnel  may  show  they  are  qualified  on  paper,  defense  officials  still  attest 
that  the  best  training  is  on-the-job  training,  for  which  the  seasoned  contracting  personnel 
have  little  time  in  which  to  engage.  More  value  needs  to  be  placed  on  the  importance  of 
diverse  education  if  the  goal  of  best  value  is  to  be  achieved. 
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F.  NECESSITY  FOR  ACQUISITION  EDUCATION 

The  tradeoff  process  is  not  necessarily  flawed,  but  most  would  agree  that  the 
education  of  those  using  the  process  is  lacking,  therefore  resulting  in  inappropriate 
practice  of  the  process.  GAO  reported  in  the  11-8  Report,  Enhanced  Training  Could 
Strengthen  DODs  Best  Value  Tradeoff  Decisions, 

DOD  officials  also  noted  that  the  complexity  of  the  tradeoff  process 
increases  the  risk  of  bid  pro  tests...  for  example,  GAO  found  that  15  of  the 
88  contracts  awarded  using  a  best  value  tradeoff  process  reviewed  were 
protested  to  GAO,  resulting  in  4  cases  in  which  DOD  determined  that  it 
failed  to  adhere  to  the  solicitations’  requirements.  (GAO,  2010,  p.  1) 

Protests  are  undesirable  for  acquisition  personnel  as  contracts  (and  thus  the  entire 
project)  cannot  move  forward  and  focus  must  be  shifted  to  defending  source  selection 
decisions.  The  acquisition  teams  attempting  to  utilize  best  value  are  either 
misunderstanding  the  stipulations  necessary  to  utilize  best  value,  or  are  finding  late  in  the 
acquisition  that  their  evaluation  methodology  was  chosen  incorrectly.  Lack  of  forward 
planning  and  careful  prioritization  of  requirements  can  lead  to  the  need  to  change  section 
M  after  offers  have  been  received,  which  can  lead  to  protest. 

The  need  for  education  amongst  the  acquisition  workforce  is  apparent,  but  the 
method  of  delivering  that  education  is  debatable.  Some  consider  on-the-job  training  to  be 
one  of  the  most  essential  means  of  gathering  sound  source  selection  decision-making 
skills,  but  that  process  takes  years  and  the  valuable  time  of  seasoned  professionals.  The 
most  frequently  employed  option  for  defense  acquisition  education  is  from  DAU.  There 
are  five  core  DAU  courses  regarding  source  selection  evaluations: 

•  CLC  007 — Contract  Source  Selection 

•  CON  121 — Contract  Planning 

•  CON  200 — Business  Decisions  for  Contracting 

•  CON  280 — Source  Selection  and  Administration  of  Service  Contracts 

•  CON  334 — Advanced  Contingency  Contracting  Officer’s  Course 
(GAO,  2014b,  p.  19-20) 

It  should  be  noted  that  while  these  courses  are  required  for  contracting  personnel, 
the  acquisition  team  is  not  comprised  entirely  of  contracting  personnel.  Other  acquisition 
personnel,  such  as  program  managers,  technical  managers,  financial  managers,  are  not 
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necessarily  required  to  take  these  courses.  What  training  is  provided  to  other  functional 
areas  of  the  acquisition  workforce  working  on  source  selection  boards  and  making  high- 
level  contracting  decisions?  The  potential  lack  of  training  for  non-contracting  personnel 
could  be  cause  for  concern. 

Each  course  contains  curriculum  that  specifically  benefits  a  source  selection  team. 
These  courses  provide  quality  information  from  seasoned  acquisition  professionals  and 
build  a  foundation  for  acquisition  knowledge.  While  DAU  training  is  essential  to  build  a 
baseline  for  acquisition  education,  on-the-job  training  supplements  and  fills  in  gaps  with 
real-world  experience.  The  increased  workload  of  acquisition  professionals  and 
bureaucracy  of  requirements  for  the  acquisition  process  leads  to  an  overworked, 
undereducated  staff  that  must  make  award  with  limited  resources.  In  essence,  the  problem 
is  not  the  quality  of  the  education,  but  rather  the  limited  time  allotted  for  acquisition 
professionals  to  break  away  from  their  jobs  for  in-residence  course  time.  Just  as  rare  is 
the  time  for  seasoned  professionals  to  impart  knowledge  on  the  newer  workforce. 

Beyond  the  established  need  for  education,  the  challenging  environment  in  which 
defense  acquisition  professionals  must  operate  adds  immeasurable  stress  to  the 
procurement  process.  Best  value  proves  to  be  helpful  in  unstable  environments,  such  as 
war  zones  where  past  performance  and  technical  capability  are  factors  of  greater 
importance,  and  for  construction  projects  where  safety  and  schedule  requirements  are 
paramount.  LPTA,  however,  is  being  used  more  frequently  for  purchase  of  standard 
commercial  products,  and  are  proven  to  require  fewer  resources  to  meet  award  timelines. 
To  mandate  that  one  method  must  be  used  over  another  invites  a  myriad  of  examples  that 
prove  the  opposite.  Defense  professionals  must  become  seasoned  in  the  rules  of 
acquisition  in  order  to  navigate  in  challenging  environments.  The  tradeoff  process  has 
been  determined  to  be  “far  more  time-consuming  than  other  approaches,” 
“administratively  burdensome,”  and  due  to  the  extensive  time  of  the  acquisition  process, 
it  is  “challenging  to  keep  the  same  acquisition  team  together  for  an  entire  procurement” 
(GAO,  2010,  p.  18).  Contracting  Officers  are  also  nervous  about  the  increased  risk  of  a 
bid  protest  due  to  the  complexity  of  the  tradeoff  process,  which  creates  an  environment  of 
caution  and  wariness  for  trying  new  contracting  approaches. 
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There  is  a  thirst  for  standardization  in  the  tradeoff  process,  and  further,  a  desire 
for  knowledgeable  personnel  who  can  provide  sound  business  judgment.  Acquisition 
professionals  find  that  “developing  non-cost  factors  that  meaningfully  discriminate 
between  offers  is  a  challenging  part  of  the  tradeoff  process”  (GAO,  2010,  p.  17). 
Therefore,  just  getting  the  evaluation  factors  defined  is  strenuous  for  an  inexperienced 
workforce.  Furthermore,  “the  absence  of  meaningful  non-cost  discriminators  can  result  in 
offerors  receiving  equal  scores  on  the  factors  that  were  identified  as  being  significantly 
more  important  than  price”  (p.  17).  So  if  the  tradeoff  process  was  selected  because  the 
non-cost  factors  were  deemed  more  important  than  price,  but  then  those  non-cost  factors 
were  not  meaningful,  then  the  resulting  acquisition  method  is  not  only  inappropriate  for 
LPT  A  but  also  poorly  designed  for  a  an  appropriate  tradeoff.  If  the  acquisition  team  was 
entering  an  acquisition  well-equipped  with  knowledge  of  the  requirement,  sound  business 
judgment,  and  a  wealth  of  experience  and  confidence  in  performing  a  source  selection, 
there  is  greater  chance  for  a  successful  contract  and  beneficial  outcomes. 

In  order  to  give  a  contract  the  chance  to  be  successful,  the  acquisition  team  must 
be  prepared  and  confident  in  their  knowledge  of  contracting.  For  an  organization  to  foster 
an  appreciation  for  education  and  expertise  in  best  value  contracting,  there  must  be  an 
environmental  shift  to  put  best  value  contracting  on  the  forefront  of  policymaking 
decisions  and  DOD  priorities.  The  DOD  is  making  efforts  to  provide  guidance  to  best 
utilization  of  best  value  contracting  methods,  but  the  impact  of  these  measures  will  take 
time  to  evaluate. 

G.  GOVERNMENT  EFFORTS  TO  REFORM  BEST  VALUE 

In  April  2015,  the  third  installment  of  the  Better  Buying  Power  series  was 
released.  Within  the  release  of  version  3.0,  under  the  “Incentivize  Innovation  in  Industry 
and  Government”  initiative  (Kendall,  2015c,  p.  2),  there  is  a  goal  to  “provide  clear  and 
objective  ‘best  value’  definitions  to  industry”  (p.  18).  This  general  guidance  states: 

This  BBP  3.0  initiative  builds  on  the  work  started  in  BBP  2.0  to  provide 
industry  with  information  on  the  value,  in  monetary  terms,  of  higher  levels 
of  perfonnance  than  minimally  acceptable  or  threshold  levels.  Without 
this  infonnation,  the  default  position  will  be  to  bid  to  the  lowest 
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acceptable  level  of  performance.  With  this  information,  industry  will 
know  what  the  competitive  effect  of  offering  higher  performance  will  be 
and  can  bid  accordingly.  (Kendall,  2015c,  p.  18) 

Government  is  clearly  working  towards  revitalizing  the  implementation  of  quality 
best  value  practices,  although  industry  may  not  be  aware  of  these  strides  until  evidence  of 
improvement  is  visible  from  active  acquisition  teams.  Kendall  followed  up  with  a  memo 
in  May  2015  titled  the  Appropriate  Use  of  LPTA  Source  Selection  Process  and 
Associated  Contract  Type,  which  not  only  anticipates  a  shift  from  use  of  Firm  Fixed  Price 
or  Time  &  Materials  to  a  Cost  Plus  Fixed  Fee  Level  of  Effort  contract  type,  but  more 
importantly  emphasizes  that  the  DOD  is  not  utilizing  LPTA  in  its  “limited  place  in  the 
source  selection  ‘best  value’  continuum”  (Kendall,  2015a,  p.  1).  Kendall  identifies  how 
DOD  has  been  inappropriately  utilizing  LPTA  for  acquisitions  too  complex  or  with  less- 
defined  requirements  that  would  benefit  from  an  innovative  contactor  solution,  and  as  a 
result  “can  miss  an  opportunity  to  secure  an  innovative,  cost-effective  solution  to  meet 
Warfighter  needs  to  help  maintain  our  technological  advantage”  (p.  1-2).  Industry  has 
responded  already  to  this  memorandum  with  favor,  as  LPTA  has  historically  denied 
industry  the  chance  to  propose  an  inventive  solution  for  somewhat  murkier  requirements. 
Bob  Lohfeld  responded  to  this  memorandum  in  an  April  2015  Washington  Technology 
article  stating  that  for  technical  service  and  solution  procurements,  “increased 
perfonnance  generally  provides  additional  value  to  the  government,  and  the  government 
should  strive  to  achieve  an  innovative,  cost-effective  solution  to  meet  mission  needs  and 
maintain  our  technology  advantage”  (Lohfeld,  2015,  para.  5).  DOD  and  industry 
therefore  agree  that  unless  acquisition  teams  strive  harder  to  properly  utilize  LPTA  where 
appropriate  and  tradeoff  where  innovation  is  key,  the  U.S.  will  gradually  lose  the 
technological  edge  in  worldwide  defense.  If  DOD  seeks  to  gain  the  best  value  for  the 
Soldier  where  it  counts,  proper  requirements  development  and  best  value  method 
employment  are  critical  steps  in  achieving  that  chance  for  technological  superiority. 

Sunnounting  the  hurdles  that  divide  government  and  industry  is  a  challenge  that 

can  be  approached  by  identifying  industry’s  perceptions  and  criticisms  of  the  government 

procurement  process.  While  the  DOD  is  making  strides  to  reform  the  best  value 

contracting  methodology,  there  are  skeptical  audiences  that  would  suggest  other  ways  to 
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reform  parts  of  the  process.  The  next  section  will  discuss  a  sample  of  industry  criticisms 
of  government  procurement  and  solicitation  planning,  and  the  source  selection  evaluation 
phase. 

H.  INDUSTRY  CRITICISMS  OF  SOURCE  SELECTION  PRACTICES 

Industry  and  government  often  have  conflicting  priorities  that  overtake  the 
discussions  and  goals  of  the  mission.  While  government  is  responsible  for  being  good 
stewards  of  the  taxpayer  dollars  and  the  best  interests  of  the  public,  industry  is 
responsible  for  the  concerns  of  their  stakeholders,  which  often  includes  profit. 
Conflicting  priorities  often  result  in  burden  on  the  contracting  process,  but  can  also  reveal 
inherent  issues  with  the  system  that,  if  addressed,  can  facilitate  the  contracting  process 
into  obtaining  best  value  and  benefit  for  industry  and  government  alike. 

1.  Procurement  Planning  Phase 

The  two  sides  of  the  fence  on  which  industry  and  government  reside  are  filled 
with  differing  priorities  that  result  in  hearty  proposal  evaluations  when  both  are  willing  to 
discuss.  Cox  in  his  article  “Finding  a  Bargain  for  Government  Buyers  Shouldn’t  Be  a 
Crime”  from  Government  Executive  magazine  argues  this  point  citing  that  while  industry 
complains  that  use  of  LPT  A  takes  a  bite  out  of  costs  (in  terms  of  proposal  development 
and  aiming  to  underbid  to  stay  competitive) — when  in  reality  it  mostly  just  takes  a  bite 
out  of  industry  profits — that  “bite”  is  nothing  compared  to  the  overall  budget  cuts 
happening  in  DOD  (Cox,  2015).  While  in  a  generous  world  the  government  would  work 
harder  to  reward  the  contractor  with  higher  profits,  or  industry  would  relinquish  profits  in 
the  name  of  the  taxpayer  dollar,  the  reality  is  that  neither  party  is  willing  to  financially 
give  away  the  farm,  resulting  in  a  contentious  debate  over  costs.  Contracting  Officers 
work  hard  to  establish  fair  and  reasonable  pricing  from  offerors  to  ensure  that  all  parties 
are  experiencing  a  fair  business  relationship,  but  the  perception  amongst  industry  remains 
skeptical  of  the  government’s  practices. 

Cox  goes  on  to  make  the  classic  argument  that  if  industry  and  government  worked 
together  to  develop  accurately  written  requirements  documents,  then  industry  could 

propose  accordingly  and  the  government  would  get  what  it  asks  for  (Cox,  2015). 
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However,  when  requirements  are  poorly  defined  and  LPT  A  is  utilized,  the  resulting 
proposals  do  not  necessarily  hit  the  “technically  acceptable”  mark  for  what  the 
government  desires.  As  a  result,  either  protests  ensue  or  the  government  moves  forward 
and  receives  inadequate  perfonnance  with  no  room  to  recoup  expenses. 

Another  widely  accepted  opinion  Cox  demonstrates  is  that  industry  insists  that 
tradeoff  is  more  beneficial  to  the  government,  but  that  insistence  is  really  a  perfonnance 
in  which  industry  tells  the  government  what  to  buy  (Cox,  2015).  Using  this  reasoning,  the 
government  is  not  incentivized  to  correct  poorly  defined  requirements,  and  instead  uses 
industry  to  develop  the  requirements  and  deliver  the  requirements — at  a  premium.  While 
all  can  agree  that  there  must  be  a  meeting  of  the  minds  in  order  for  both  sides  of  the  fence 
to  speak  the  same  language  and  reach  consensus  on  contract  perfonnance  requirements, 
the  contentious  nature  of  the  cost-saving  priorities  on  both  sides  is  a  challenging  obstacle 
to  overcome. 

In  a  2014  National  Defense  Magazine  article,  Stew  Magnuson  refers  to  a 
conference  where  the  president  of  a  satellite  company  stated  that  government  “is 
exchanging  new  applications  and  value  added  services  for  lower  prices”  (Magnuson, 
2014,  para.  3).  The  president  goes  on  to  say  that  “you  may  have  written  the  best  proposal 
of  your  life... but  it  will  never  be  read  by  the  government  because  of  that  LPTA 
requirements... it  is  a  shame  because  there  is  no  room  for  innovation  when  you  do  that” 
(Magnuson,  2014,  para.  6).  Magnuson  quotes  in  his  article  a  president  of  a  satellite 
communications  company  as  saying  “the  lowest  bid  gets  its  proposal  evaluated  to  see  if  it 
meets  the  technical  requirements... if  it  does,  none  of  the  others  are  considered  or  even 
read”  (Magnuson,  2014,  para.  6).  Any  knowledgeable  contracting  officer  would  refute 
that  accusation  and  insist  that  all  proposals  are  reviewed;  but  this  does  not  reduce  that 
stigma  among  industry  if  time  after  time  the  lowest  bidder  is  found  to  be  the  winner.  This 
increased  tension  between  government  and  industry,  as  well  as  an  increasing  fear  of 
protest  amongst  acquisition  professionals,  is  contributing  to  higher  walls  being 
constructed  between  the  two  entities.  An  early  mitigation  for  this  tension  is  to  invite 
industry  to  discussion  regarding  the  requirements  development  to  get  them  involved 
earlier  in  requirements  discussions. 
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There  is  a  very  clear  need  for  stronger  industry  involvement  in  requirements 
development  and  market  research.  This  is  a  beneficial  step  not  in  the  sense  that  industry 
is  writing  the  requirements  on  which  they  would  bid,  but  government  would  get  a  better 
idea  for  what  is  most  important  in  their  acquisition  if  they  discuss  the  requirement  openly 
with  industry  prior  to  release  of  the  final  RFP.  The  acquisition  team  should  be  performing 
thorough  market  research  to  have  insight  regarding  the  capabilities  of  industry  and 
concentration  of  certain  skills  and  practices,  before  determining  if  there  is  even  room  for 
innovation  in  an  acquisition.  If  requirements  are  well-developed  and  cost  estimates  are 
firm  and  display  all  aspects  of  potential  costs,  then  reviewing  and  evaluating  proposals 
should  be  relatively  simple,  as  all  data  involved  is  reliable. 

In  the  late  1980s  and  early  1990s,  when  government  was  in  the  throes  of  reducing 
costs  and  downsizing  bureaucracy  and  personnel,  best  value  may  not  have  been 
implemented  properly  by  source  selection  teams,  thereby  resulting  in  a  negative  view  of 
tradeoff  practices  in  industry.  In  a  hearing  before  the  Subcommittee  on  Legislation  and 
National  Security  of  the  Committee  on  Government  Operations  in  May  1993,  Stephanie 
Biddle,  President  of  the  Computer  &  Communications  Industry  Association  (CCIA), 
stated  in  response  to  the  recently  approved  Federal  Acquisition  Improvement  Act  of 
1993: 


Industry’s  greatest  concern  with  best  value  is  not  that  we  don’t  think  the 
government  ought  to  get  it;  we  think  that  the  problem  is  the  very  loose 
evaluation  criteria  that  characterize  these  acquisitions  and  that  may,  in 
fact,  produce  an  award  that  is  not  based  on  any  meaningful  competition. 
The  evaluation  criteria  in  best  value  procurements  usually  do  not  tell 
vendors  critical  infonnation  regarding  the  relative  importance  that  the 
agency  places  on  price  and  technical  factors.  A  typical  best  value 
procurement  solicitation  may  say  little  more  than  technical  considerations 
are  more  important  than  cost.  But  the  agency  does  not  give  any  indication 
regarding  the  extent  to  which  technical  considerations  will  outweigh  price. 
Put  into  tenns  that  you  and  I  can  understand,  it  is  like  saying,  “I  don’t 
know  if  I  want  to  buy  a  Cadillac  that  is  comfortable  or  I  want  to  buy  a 
little  less  expensive  car  that  gets  good  gas  mileage.”  You  have  to  have 
some  idea  what  you  want  to  buy  before  you  spend  hundreds  of  millions  of 
dollars.  (Biddle,  1993,  p.  121) 
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This  scathing  review  of  government  perfonnance  provides  a  concise  snapshot  of 
how  industry  viewed  government’s  grasp  on  requirements  development  and  its  shoddy 
application  towards  tradeoff.  Biddle  is  emphasizing  with  the  Cadillac  example  that  even 
if  the  Cadillac’s  requirements  are  well-defined,  if  there  are  no  properly  described 
evaluation  factors  from  which  industry  can  understand  where  government’s  priorities  lie, 
then  only  half  of  the  necessary  information  is  present.  As  with  Cox’s  previously  reviewed 
accusation  that  government  relies  too  heavily  on  industry’s  ability  to  write  requirements, 
Biddle  confirms  that  even  if  the  requirements  are  well  written  for  that  Cadillac,  the 
evaluation  factors  must  coincide  with  the  government’s  priorities  for  those  requirements 
to  come  to  fruition  with  a  capable  Contractor.  Given  the  strides  that  DAU  courses, 
defined  policies  and  better  understanding  of  how  to  develop  and  implement  evaluation 
factors  for  source  selections  (due  to  the  Federal  Acquisition  Reform  Act  of  1993)  have 
made,  this  rift  has  been  addressed  if  not  somewhat  closed.  While  industry  and 
government  may  still  disagree  on  the  requirements  themselves,  there  are  paths  forward  to 
opening  communication  between  both  parties  via  properly  prioritized  evaluation  factors. 

2.  Solicitation  Preparation  Phase 

Lohfeld  argues  in  his  2012  Washington  Technology  article  “Will  low-price 
contracting  make  us  all  losers?”  that  LPTA  is  not  being  used  effectively,  therefore  LPTA 
is  not  an  effective  evaluation  method  (Lohfeld,  2012).  Given  the  recent  increased  use  of 
LPTA,  government  needs  to  be  making  strides  in  using  LPTA  effectively.  The 
government’s  take  on  requirements  needs  to  shift  from  telling  industry  how  they  are  to 
perform  without  considering  their  input,  and  move  to  engaging  industry  in  requirements 
development  early  in  the  process  (Lohfeld,  2012).  When  both  parties  understand  the  end 
goal,  and  work  together  to  write  requirements  that  speak  to  that  level  of  effort,  the  RFPs 
could  more  accurately  depict  what  the  government  is  seeking  to  acquire.  By  engaging  in 
discussions  early,  government  is  less  likely  to  misstep  and  inadvertently  write 
requirements  too  vague  for  which  too  many  vendors  are  deemed  technically  acceptable. 

Scott  Calisti  suggests  in  his  DAU  AT&L  magazine  article,  “Lowest  Price 
Technically  Acceptable:  Why  All  the  Debate?”  that  industry  continues  to  be 
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unwelcoming  of  LPT  A  as  a  means  of  evaluation,  insisting  that  while  cost  is  being  driven 
down,  so,  too,  is  quality,  stating  that  “LPTA  drives  us  to  only  a  Tow  cost,  low  quality’ 
solutions”  (Calisti,  2015,  p.  17).  Industry  insists  LPTA  “stifles  innovation  and  squeezes 
corporate  margins  due  to  downward  pressure  on  price”  (p.  17),  which  ultimately  “causes 
perfonnance  innovators  to  depart  the  market  and  reduce  the  quality  of  goods  and  services 
provided”  (p.  17).  While  these  observations  are  limited  in  scope,  there  is  a  compelling 
argument  for  the  detractions  of  LPTA.  These  negative  misnomers  are  rooted  in  the 
misuse  of  LPTA  by  the  government.  LPTA  is  not  being  used  appropriately,  and  is  driven 
so  by  the  regulations,  policy,  guidance,  and  training  that  have  misled  acquisition 
professionals.  Again,  if  the  requirements  are  well-defined  and  the  risk  of  unsuccessful 
contract  perfonnance  is  minimal,  and  price  has  been  detennined  to  be  the  only  deciding 
factor  for  best  value,  then  LPTA  is  the  optimal  choice  for  evaluation.  The  five  DAU 
courses  that  assist  in  training  contracting  professionals  in  source  selection  skills  and  other 
agency  mediums  for  training  adhere  to  the  appropriate  use  of  LPTA,  but  the  skills  are 
either  not  being  translated  appropriately  into  use  or  are  being  misunderstood. 

Calisti  continues,  “Industry  contends  that  quality  solutions  and  LPTA  are 
mutually  exclusive  and  we  cannot  buy  quality  goods  and  services”  (Calisti,  2015,  p.  20). 
If  the  government  has  determined  their  requirements  and  are  certain  where  the  dollar  is 
limited  on  providing  further  returns,  then  industry  is  forced  to  offer  higher  perfonnance 
for  reduced  cost  to  remain  competitive,  which  cuts  into  profits  and  is  undesirable  for 
industry.  As  long  as  government  is  adhering  to  standards  of  best  value,  and  not  simply 
employing  LPTA  because  resources  for  a  proper  tradeoff  evaluation  are  limited  or 
pressure  to  make  a  timely  award  are  overwhelming,  then  industry’s  profit  margins  are  not 
of  their  concern. 

3.  Source  Selection  Evaluation  Phase 

According  to  Robert  Graham,  in  his  2007  article  in  Contract  Management  entitled 
“Improving  Source  Selections,”  the  “conspiracy  of  hope”  occurs  when  “industry 
encouraged  to  be  overly  optimistic  in  their  offers  and  government  imposes  little  or  no 
financial  risk  to  those  who  submit  such  offers”  (Graham,  2007,  p.  28).  The  conspiracy  is 
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that  they  can  underbid  early,  and  upon  contract  award  begin  to  ramp  up  the  costs  via 
modifications  and  change  proposals  previously  left  undiscussed  or  not  considered  in  the 
Independent  Government  Cost  Estimate  (IGCE).  Graham  refers  to  the  IGCE  as  the  Most 
Probable  Cost  (MPC),  as  it  goes  beyond  the  traditional  IGCE  to  capture  all  possible  costs, 
not  just  the  minimum  standard  or  average  costs  the  government  desires  to  spend.  Using 
the  MPC  indicates  that  the  government  has  thoroughly  considered  the  probabilities  of 
costs  based  on  well-developed  requirements  (p.  32). 

The  importance  of  cost  risk  is  important  to  note  as  a  separate  factor  that  can 
enhance  proposal  risk  evaluation.  Graham  recommends  determining  a  high-confidence 
cost  estimate  earlier  in  the  source  selection  (as  in,  going  beyond  the  IGCE)  and  elevating 
the  cost  risk  evaluation  factor  to  a  more  important  position  in  the  source  selection 
evaluation  (Graham,  2007,  p.  32),  meaning  to  separate  and  clearly  define  cost  risk  and 
proposal  risk.  Right  now  the  source  selection  panels  are  confusing  the  two;  “cost  risk  is 
meant  to  be  a  measure  of  the  degree  of  difference  between  an  offeror’s  proposed  cost  and 
the  government’s  developed,  most  probable  cost — not  an  identification  of  proposal  risk” 
(p.  32).  Therefore,  if  the  government  has  a  high-confidence  cost  estimate  developed  early 
in  the  acquisition  process,  then  when  it  comes  time  to  evaluate  the  proposals  a  clear  price 
differential  can  be  defined  faster,  thereby  establishing  the  cost  risk.  “The  cost-risk  rating 
assesses  the  degree  to  which  an  offerors ’s  cost  proposal  for  the  contract  line  items 
compares  with  the  government’s  computed  MPC  for  the  same  items”  (p.  33).  This 
practice  would  reduce  the  “conspiracy  of  hope”  amongst  industry,  but  requires  a  cultural 
and  educational  change  amongst  government  personnel  to  understand  these  evaluation 
factors  and  the  significance  on  the  ability  to  make  a  decision  based  on  sound  business 
judgment.  According  to  Graham, 

These  recommendations  will  establish  the  expectation  at  the  outset  of 
source  selection  that  in  order  to  evaluate  the  offeror’s  cost  proposal  and 
develop  the  government’s  probable  cost,  more  work  will  be  necessary 
(both  before  competitive  range  detenninations  and  during  the  ensuing 
discussion  period)  for  the  cost  evaluation  team  to  assess  cost  risk  and 
conduct  meaningful  discussions  with  all  offerors  to  ensure  maximum 
agreement.  (Graham,  2007,  p.  33) 
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Appropriate  use  of  cost  risk  during  evaluations  could  effectively  change  how  cost  savings 
are  recognized  during  pre-award. 

Another  means  of  finding  cost  savings  and  improving  best  value  is  to  reconsider 
what  qualified  as  “technically  acceptable.”  Lohfeld  opines  in  his  2012  article  that 
“technically  acceptable”  should  be  a  level  of  performance  that  is  harder  to  achieve.  He 
states  that  “setting  the  standards  too  low  will  let  marginally  acceptable  bidders  become 
candidates  for  contract  award.  These  are  the  same  bidders  who  would  normally  have  been 
weeded  out  in  best-value  tradeoff  procurements”  (Lohfeld,  2012,  para.  8).  If  all  proposing 
vendors  meet  the  minimum  requirements,  then  those  requirements  were  not  challenging 
enough  to  eliminate  potentially  underperfonning  vendors.  If  every  vendor  makes  it 
through  the  original  round  of  eliminations  (even  the  vendors  who  only  marginally  pass 
for  acceptable),  then  the  only  factor  on  which  they  would  be  evaluated  is  price,  which  is 
not  a  responsible  business  decision  when  the  perfonnance  could  suffer  tremendously.  The 
government  cannot  give  additional  favor  to  a  vendor  that  well  exceeds  a  requirement  over 
a  vendor  that  only  marginally  passed;  that  vendor  is  considered  at  the  same  level  as  a 
marginally  acceptable  vendor.  Therefore,  raising  the  bar  and  developing  requirements 
that  are  more  difficult  to  achieve  could  be  an  effective  way  to  utilize  LPTA. 

Lohfeld  continues  to  criticize  FAR  15.305(a)(2),  the  section  of  the  FAR  dedicated 
to  proposal  evaluation  specifically  for  past  performance,  stating  that, 

When  applying  LPTA  criteria  to  past  perfonnance  evaluations,  an  offeror 
without  a  record  of  relevant  past  performance  or  for  whom  information  on 
past  performance  is  not  available  or  is  so  sparse  that  no  meaningful  past 
performance  rating  can  be  reasonably  assigned,  the  offeror  may  not  be 
evaluated  favorably  or  unfavorably  on  past  perfonnance  (see  FAR  15.305 
(a)(2)(iv)).  Therefore,  the  offeror  shall  be  detennined  to  have  unknown 
past  perfonnance.  In  the  context  of  acceptability/unacceptability,  unknown 
shall  be  considered  acceptable.  (Lohfeld,  2012) 

This  pointed  criticism  from  industry  indicates  that  industry  feels  the  past  perfonnance 
evaluation  factor  handicaps  the  evaluation  process  by  lowering  the  technical  acceptability 
bar  for  past  perfonnance  reviews  for  vendors  with  unknown  past  performance.  If 
“unknown”  is  considered  “acceptable,”  the  acceptability  rate  for  vendors  in  an  acquisition 
could  be  unintentionally  raised,  causing  a  potentially  unfair  advantage  for  a  vendor  that 
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should  not  have  been  found  acceptable.  This  problem  may  wane  on  its  own  as  DOD 
personnel  improve  on  CPARS  reporting  rates,  but  should  be  considered  for  new 
companies  entering  DOD  contracting. 

Multiple  opinions  insist  that  awarding  LPTA  with  emphasis  on  the  low  price 
nearly  guarantees  poor  perfonnance  from  the  contractor.  Per  Lohfeld’s  article, 

The  satisfaction  of  having  awarded  to  the  lowest  price  offer  can  soon  be 
overshadowed  by  the  burden  of  poor  contractor  performance.  Slow 
contract  staffing  is  an  early  indicator  of  contractor  performance  problems, 
followed  by  marginal  technical  accomplishment,  late  deliveries,  and 
ultimately  cost  overruns.  In  the  long  run,  the  lowest  bidder’s  performance 
reputation  suffers,  and  the  government  is  criticized  for  cost  and  schedule 
overruns  and  for  failing  to  manage  their  program  correctly.  In  the  end, 
everyone  loses.  (Lohfeld,  2012,  para.  18) 

While  this  is  merely  an  opinion,  it  is  a  perception  from  industry  that  LPTA  is  an  indicator 
that  government  does  not  value  innovative  solutions,  nor  high-quality  performance.  If 
industry  has  these  perceptions,  there  may  be  indication  of  a  self-fulfilling  prophecy 
where,  regardless  if  this  opinion  is  true,  the  mere  existence  of  that  opinion  in  industry 
could  make  the  hypothetical  outcome  a  reality.  The  contractor  decides  their  quality  of 
perfonnance  on  a  contract;  if  there  is  an  underlying  perception  by  that  contractor  that 
LPTA  means  lower  quality  perfonnance,  there  may  be  an  increased  likelihood  that  lower 
quality  performance  naturally  occurs  as  a  result.  This  can,  however,  be  mitigated 
throughout  the  life  of  the  contract  by  the  active  monitoring  and  surveillance  performed  by 
the  government  Contracting  Officer’s  Representative  (COR)  appointed  to  the  contract. 
With  the  help  of  the  COR,  some  common  ground  may  be  found  between  government  and 
industry  perceptions  of  how  best  value  is  being  obtained  by  the  government,  through  the 
process  of  reporting  past  performance  infonnation. 

I.  CONTRACTOR  PAST  PERFORMANCE  INFORMATION 

In  2009,  the  Office  of  Federal  Procurement  Policy  (OFPP)  release  a  memorandum 
titled  “Improving  the  Use  of  Contractor  Performance  Information”  that  described  the 
need  for  a  system  that  would  hold  contractors  accountable  for  past  performance.  At  that 
time,  agencies  used  “fragmented  methods  to  collect  and  maintain  contractor  performance 


30 


information”  (OFPP,  2009,  p.  1),  including  manual  performance  reviews  that  are  kept  in 
the  paper  contract  file,  or  were  utilizing  “internal  data  systems  that  are  not  available  to 
acquisition  officers  outside  that  agency”  (p.  1).  The  limited  distribution  of  perfonnance 
information  handicapped  the  acquisition  system  in  that  future  acquisition  teams  were 
unable  to  retrieve  the  past  perfonnance  information  that  could  be  critical  in  evaluating  a 
contractor’s  performance  history.  In  this  memorandum,  the  FAR  was  officially  changed 
effective  July  1,  2009  to  require  agencies  to  submit  an  electronic  record  of  contractor 
perfonnance  in  the  Past  Performance  Information  Retrieval  System  (PPIRS),  which  is  a 
web-based  system  that  serves  as  “the  single,  government-wide  repository  for  contractor 
perfonnance  information”  (p.  1). 

In  2004,  the  report-building  system  CPARS  was  developed  by  the  Navy  and 
became  the  mandatory  system  for  capturing  past  performance  in  the  DOD  (Guidance, 
2014).  CPARS  is  a  web-enabled  system  designed  to  host  a  history  of  contractor 
perfonnance  on  specific  contracts,  as  submitted  by  the  receiver  of  the  contractor’s 
supplies  or  services.  CPARS  is  a  system  of  processes  that  assists  the  assessing  official  in 
assembling  a  review  of  a  contractor’s  performance  based  on  various  limited  elements, 
including  quality  of  the  product/service,  schedule  quality,  ability  to  control  cost,  quality 
of  business  relations,  utilization  of  small  business,  and  key  personnel  management,  as 
relevant  to  the  specific  contract  (Guidance,  2014). 

The  reporting  process  is  designed  to  be  fair  and  objective.  The  assessing  official 
or  contracting  officer  assembles  a  submission  to  the  CPARS  system  that  provides  an 
evaluation  of  the  contractor’s  perfonnance  for  the  various  elements.  The  draft  report 
created  in  CPARS  is  sent  to  the  contractor,  who  is  allowed  30  days  to  respond  to  the 
report  and  provide  comments  or  rebuttals  for  the  infonnation.  The  contractor’s  response 
is  reviewed  by  the  assessing  official  and  is  given  time  to  revise  the  original  assessment 
and  narrative  accordingly;  if  there  is  a  disagreement  with  the  assessment,  a  government 
official  one  level  above  the  assessing  official  organizationally  will  review  and  finalize  the 
assessment  (Guidance,  2014,  p.  25-26). 

Prior  to  establishment  of  major  past  perfonnance  review  systems  (like  PPIRS), 

past  perfonnance  information  needed  for  source  selection  review  boards  was  so 

31 


unreliable  that  Source  Selection  Evaluation  Boards  (SSEBs)  met  the  delivered  data  with 
skepticism  and  distrust.  The  unreliability  of  that  data  was  based  on  lack  of  standardized 
reporting  measures  for  documenting  and  filing  past  performance  data.  Further,  when  it 
was  finally  launched,  PPIRS  lacked  oversight  and  management.  Lack  of  accountability, 
unclear  system  metrics  and  tools,  and  varying  definitions  for  rating  factors  contributed  to 
overall  frustration  with  the  past  performance  reporting  process.  The  first  hurdle  is  to  get 
data  on  past  perfonnance,  the  second  is  to  ensure  that  the  data  is  reliable,  and  the  third  is 
to  use  that  data  appropriately  to  make  source  selection  decisions;  if  the  data  cannot  be 
trusted,  then  it  cannot  be  used,  and  an  entire  evaluation  factor’s  opportunity  is  wasted. 

Annie  Meeks  discusses  the  concept  of  oversight  in  the  post-award  phase  of 
contracting  in  her  2011  NCMA  article,  “The  Buck  Stops... Where?:  Post-Award 
Accountability  for  Large  Service  Contracts.”  She  argues  that  without  sufficient  contract 
oversight,  knowledge  is  lost  and  minor  problems  go  undetected  and  become  major 
problems  if  not  crises.  Meeks  reviews  the  three  necessary  levels  of  oversight  for  thorough 
contract  management,  maintained  by  both  government  and  contractor: 

•  Quality  Control:  the  plan  designed  by  the  Contractor,  enforced  by  the 
Contractor,  to  review  quality  of  performance; 

•  Quality  Assurance:  the  plan  for  review  of  Contractor  performance, 
enforced  by  the  Government,  to  review  quality  of  Contractor  perfonnance; 

•  Quality  Oversight:  the  plan  for  review  of  Government  evaluation  of 
Contractor  performance,  enforced  by  the  Government,  to  review  how  the 
Government’s  processes  are  being  enforced  to  review  Contractor 
perfonnance. 

(Meeks,  201 1,  p.  37) 

When  both  the  government  and  the  contractor  are  held  responsible  for  contract  oversight, 
fewer  details  are  missed  in  perfonnance  evaluations,  quality  is  maintained  in  accordance 
with  contract  requirements,  and  reporting  via  the  necessary  channels  and  systems  is  used 
to  benefit  future  evaluations. 

In  all  best  value  evaluations,  evaluation  factors  must  clear,  measurable,  given 
weight,  and  understood  by  all  parties.  When  the  government  was  evaluating  contractor 
perfonnance  and  either  party  did  not  understand  that  which  was  being  evaluated  or  the 
factors  being  utilized,  resources  would  be  wasted  in  the  discussions  over  what  was  being 
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entered  into  official  record  (if  anything  was  being  entered).  Without  oversight  for  a 
system  that  maintains  past  performance  information,  there  is  no  incentive  to  create 
records  of  past  performance.  Without  past  perfonnance  information,  future  source 
selection  evaluation  boards  go  without  pertinent  information  to  make  quality  best  value 
decisions.  The  qualifications  of  the  COR  should  therefore  be  adequate  for  the  work  they 
are  perfonning.  The  COR  needs  not  only  to  intimate  technical  knowledge  of  the  work 
being  perfonned  to  be  able  to  adequately  convey  a  comprehensive  review  of  that 
perfonnance;  “staff  should  be  subject  matter  experts  trained  in  the  observation  of  tasks 
and  factual  measurement  techniques”  (Meeks,  2011,  p.  38).  CORs  should  also  have  the 
work  ethic  to  physically  perfonn  the  oversight  and  surveillance  responsibilities  of  COR 
to  ensure  that  evaluation  of  service  provider  work  is  thorough  and  reliable.  COR  training 
provides  appropriate  theoretical  application  of  COR  skills,  but  only  through  repetitive 
practice  and  thorough  understanding  of  performance  evaluation  factors  can  post-award 
evaluation  provide  appropriate  data  that  reflects  the  quality  desired  when  the  best  value 
processes  were  developed  during  source  selection. 

“Especially  in  this  era  of  economic  recession,  service  providers  and  government 
agencies  cannot  afford  to  waste  time,  money,  and  resources  on  failed  service  delivery 
contracts,”  (Meeks,  201 1,  p.  42)  particularly  when  the  goal  of  best  value  processes  was  to 
gain  more  outcome  without  more  dollars.  Without  proper  evaluation  of  contract 
perfonnance,  there  is  no  firm  way  of  knowing  whether  the  government  received  what  it 
paid  for.  While  the  U.S.  is  currently  emerging  from  a  recession,  the  defense  budget  cuts 
are  growing  in  size  and  as  a  result,  contract  management  teams  must  operate  with  limited 
resources.  Therefore,  once  the  contract  has  been  awarded  utilizing  the  evaluation  method 
that  best  suits  the  government’s  interest,  effective  contract  management  must  be 
perfonned  to  ensure  that  contractors  are  delivering  on  their  requirements.  Undelivered 
requirements  are  indicative  of  taxpayer  dollar  waste,  even  more  so  than  dollars  lost  in 
evaluations  and  negotiations. 

GAO  Report  09-374,  Federal  Contractors:  Better  Performance  Information 
Needed  to  Support  Agency  Award  Decisions,  suggested  that  the  Administrator  of  the 
OFPP  work  to: 
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•  Standardize  evaluation  factors  and  rating  scales  government-wide  for 
documenting  contractor  performance; 

•  Establish  policy  for  documenting  perfonnance-related  not  currently 
captured  in  PPIRS  (like  terminations,  subcontractor  management,  and 
other  pertinent  measurements); 

•  Define  government-wide  roles  and  responsibilities  for  managing  and 
overseeing  PPIRS  data; 

•  Develop  system  tools  and  metrics  for  agencies  to  use  in  monitoring  and 
managing  the  documenting  of  contractor  perfonnance; 

•  Take  appropriate  action  to  finalize  proposed  changes  to  the  FAR  that 
clarify  responsibilities  and  perfonnance  documentation  requirements  for 
contract  orders,  including  those  against  the  GSA  multiple  award  schedule. 
(GAO,  2009,  p.  21) 

OFPP  designated  CPARS  as  the  single  government-wide  system  for  entering 
evaluations,  and  by  October  2010  ah  agencies  had  transitioned  to  using  CPARS.  The 
CPARS  Guide  has  standardized  definitions  for  ratings,  which  has  thus  addressed  the 
issue  identified  in  GAO  Report  09-374  of  ratings  having  unclear  definitions. 

The  GAO  reported  an  increase  in  compliance  metrics  in  2013  in  GAO  Report  13- 
589,  DOD  Actions  to  Improve  the  Reporting  of  Past  Performance  Information,  from  56 
to  74  percent  from  October  2011  through  April  2013  (GAO,  2013,  p.  2).  While  timeliness 
of  the  reports  continues  to  be  an  issue,  the  improvement  in  report  completion  is  a  step 
forward  for  improving  past  perfonnance  tracking  for  major  contract  actions.  Source 
selection  evaluations  that  depend  on  reliable  CPARS  reporting  therefore  have  some  hope 
that  the  culture  in  the  acquisition  environment  is  gaining  stronger  reverence  for  past 
performance  infonnation,  and  will  make  stronger  strides  to  comply  with  regulations 
dictating  proper  reporting. 

It  was  reported  in  the  GAO  Report  14-707,  Contractor  Performance:  Actions 
Taken  to  Improve  Reporting  of  Past  Performance  Information,  that  “Section  806  of  the 
National  Defense  Authorization  Act  (NDAA)  for  FY  2012  required  DOD  to  develop  a 
strategy  to  ensure  that  evaluations  in  past  perfonnance  databases  used  for  making  source 
selection  decisions  are  complete,  timely  and  accurate”  (GAO,  2014a,  p.  7).  The  OFPP 
had  worked  with  the  Integrated  Award  Environment,  the  CPARS  program  office  and  an 
interagency  working  group  to  update  the  CPARS  guide  (which  was  released  in  November 
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2012),  and  the  FAR  Council  agreed  to  revise  the  FAR  in  September  2013  to  “enhance 
various  elements  of  documenting  contractor  performance,”  including: 

•  Standards  for  timeliness  (must  be  completed  within  120  days  of  the  end  of 
evaluation  period) 

•  Standards  for  completeness  (including  minimum  evaluation  factors  to  be 
completed) 

•  Assigning  responsibility  for  completeness  of  evaluations  (default  is  the 
KO,  but  the  requiring  agency  is  responsible  for  establishing  roles  and 
responsibilities  for  CPARS  reporting) 

•  Management  accountability  (agencies  are  required  to  evaluate  compliance 
and  assign  responsibility  for  CPARS  reporting) 

•  Ensuring  past  performance  submissions  are  consistent  with  award  fee 
evaluations  (award  and  incentive  fee  evaluations  are  to  be  included  as  part 
of  the  past  perfonnance  evaluations) 

(GAO,  2014a,  p.  8) 

While  the  DOD  has  generally  improved  its  CPARS  reporting,  the  rate  of 
improvement  varies  by  agency.  From  April  2013  to  April  2014,  the  DOD  only  improved 
by  7%  (from  76%  compliance  rate  to  83%  compliance  rate,  based  on  contracts  with 
dollar  thresholds  that  are  generally  higher  than  the  simplified  acquisition  threshold) 
(GAO,  2014a,  p.10).  DOD  has  the  highest  compliance  rate  amongst  the  top  ten  agencies 
that  perform  the  most  federal  contracting.  While  any  improvement  is  progress,  this  still 
means  that  for  contracts  below  the  SAT  there  is  less  stringent  regulation  for  CPARS 
reporting.  Furthermore,  the  reporting  for  contracts  above  the  SAT,  the  compliance  is  still 
only  at  83%,  meaning  that  approximately  17%  of  the  larger  contracts  have  CPARS 
reports  that  are  not  being  filed,  past  perfonnance  metrics  not  being  captured,  and  an 
immeasurable  number  of  future  source  selections  are  going  without  that  data. 

Past  perfonnance  infonnation  is  essential  to  bolster  the  probability  for  successful 
future  contracts.  Regardless  of  whether  the  reviews  are  positive  of  negative,  the 
information  is  valuable  for  source  selection  boards  looking  to  make  decisions  on  their 
future  acquisitions  based  on  the  reviews  from  contractor’s  past  work.  DOD  is  pursuing 
new  and  inventive  means  of  streamlining  vendor  selection  based  on  the  quality  of  past 
contract  performance.  In  2014  Elliot  Branch,  Deputy  Assistance  Secretary  of  the  Navy 
(Acquisition  and  Procurement)  released  a  memorandum  announcing  the  Superior 
Supplier  Incentive  Program  (SSIP): 
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“The  Superior  Supplier  Incentive  Program  is  a  pilot  program  to  incentivize 
contactor  perfonnance  by  recognizing  exceptional  contractor  conduct  in 
the  areas  of  cost,  schedule,  performance,  quality,  and  business  relations. 

The  program  assesses  three  years’  worth  of  data,  using  the  Contractor 
Perfonnance  Assessment  Reporting  System  (CPARS)” 

(Branch,  2014,  p,  1). 

In  June  2014,  the  Army  began  employing  the  SSIP  and  released  its  own  list  of  30 
vendors,  broken  out  into  three  tiers  (Tier  1  being  the  best),  and  released  the  list  to  the 
public  (Clark,  2014).  The  goal  of  the  SSIP  is  to  incentivize  contractors  to  achieve 
exceptional  perfonnance  that  will  ideally  be  reflected  in  the  CPARS  report.  The  problem 
with  the  SSIP  design  is  that  CPARS  is  cunently  under-  and  improperly  utilized  by  the 
DOD.  Success  of  the  SSIP  test  program  is  dependent  upon  reliable  CPARS  reporting.  If 
the  quality  of  CPARS  reporting  increases  over  time,  there  is  reason  to  believe  that  the 
SSIP  test  program  could  evolve  into  a  beneficial  tool  for  the  DOD. 

J.  SUMMARY 

This  chapter  contains  a  review  of  the  available  literature  relevant  to  source 
selection  methodologies  and  the  best  value  continuum,  proper  utilization  of  non-cost 
factors,  and  increasing  use  of  LPTA.  The  modern  acquisition  workforce  was  discussed  in 
terms  of  experience,  knowledge,  and  comfort  with  best  value  procurements,  as  well  as  the 
need  for  acquisition  education.  Government  best  value  contracting  reform  was  explored, 
as  well  as  criticisms  from  industry  of  government  source  selection  practices.  Past 
perfonnance  infonnation  and  its  importance  for  future  acquisition  was  discussed,  to  pave 
the  way  for  its  importance  in  the  outcomes  of  contracts  for  this  research.  The  next  chapter 
will  review  the  methodology  for  this  study,  including  a  brief  overview  of  the  contracting 
systems  used  to  gather  reports  and  data,  how  the  contract  data  was  acquired,  description 
of  the  data,  descriptive  statistics,  and  data  issues. 
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III.  METHODOLOGY 


This  section  will  explain  the  sources  and  methods  for  collecting  contractual  data 
used  in  this  study,  as  well  as  the  goal  for  establishing  the  relationship  between  the  choice 
of  source  selection  methodology  (either  lowest  price  technically  acceptable  or  tradeoff) 
and  the  resultant  measure  of  success  of  the  contract  (the  CPARS  average  rating).  This 
chapter  will  review  the  methodology  for  this  study,  including  a  brief  overview  of  the 
contracting  systems  used  to  gather  reports  and  data,  how  the  contract  data  was  acquired, 
description  of  the  data,  descriptive  statistics,  and  data  issues. 

A.  PROGRAM  EXECUTIVE  OFFICE  ENTERPRISE  INFORMATION 

SYSTEMS 

Within  the  Department  of  the  Anny  there  are  six  Secretariats,  one  of  which  is 
ASA  (ALT).  ASA(ALT)  sustains  the  mission  to  “provide  our  Soldiers  a  decisive 
advantage  in  any  mission  by  maintaining  quality  acquisition  professionals  to  develop, 
acquire,  field,  and  sustain  the  world’s  best  equipment  and  services  through  efficient 
leveraging  of  technologies  and  capabilities  to  meet  current  and  future  Army  needs” 
(“Mission  Statement,”  n.d.).  There  are  twelve  major  program  executive  offices  within  the 
Office  of  the  ASA(ALT),  one  of  which  is  PEO  EIS. 

The  PEO  EIS  is  a  $3.5  billion  Army  organization  comprised  of  34  acquisition 
programs  (“About  PEO  EIS,”  n.d.,  para.  2).  Each  program  office  employs  military, 
government  civilians  and  contractor  support.  PEO  EIS  leads  the  Army  in  development 
and  implementation  of  major  infonnation  technology  (IT)  systems  in  multiple  fields, 
including  business  management,  enterprise  resource  planning  (ERP),  biometrics, 
personnel,  training,  communications,  infrastructure,  audit  readiness,  logistics  and 
financial  management  (“About  PEO  EIS,”  n.d.,  para.  2). 

PEO  EIS  typically  purchases  COTS  major  IT  business  systems  on  a  competitive 
basis.  In  a  $3.5  billion  portfolio  of  programs,  the  contract  negotiations  can  be  extensive, 
and  the  significance  of  the  tax  dollars  spent  is  not  taken  lightly.  The  Better  Buying  Power 
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initiatives  are  employed  to  the  fullest  extent  possible,  and  gamering  the  best  value  for  the 
Soldier  is  the  goal  of  every  PEO  EIS  procurement. 

For  this  study,  contracts  awarded  for  PEO  EIS  will  be  sampled  for  evaluation  of 
the  outcomes  of  the  contract  against  the  best  value  methodology  utilized  for  evaluation 
during  competition  to  best  evaluate  how  the  Army  is  currently  utilizing  best  value 
methods  for  complex  IT  contracts. 

B.  ELECTRONIC  CONTRACTING  SYSTEMS 

The  current  contracting  environment  requires  copious  amounts  of  paperwork  and 
data  tracking.  The  best  means  for  managing  contracts,  particularly  of  the  larger  variety,  is 
to  utilize  electronic  contracting  systems  and  data  repositories,  many  of  which  limit  access 
only  to  those  with  a  need  to  know,  or  those  directly  involved  with  the  acquisition.  This 
section  will  briefly  discuss  the  individual  electronic  contracting  systems  that  were 
accessed  for  data  during  this  study. 

(1)  Virtual  Contracting  Enterprise 

The  Virtual  Contracting  Enterprise  (VCE)  is  a  suite  of  web-based  contracting 
tools  developed  and  owned  by  ACC  for  use  by  its  employees  and  their  customers  in  the 
perfonnance  of  their  daily  duties  acquiring  supplies  and  services  for  the  U.S.  Anny 
(“VCE  Mission,”  n.d.).  The  suite  of  tools  includes  a  Business  Intelligence  Metrics  & 
Reporting  (VCE-BI)  module  out  of  which  ad-hoc  reports  can  be  generated  that  compile 
contractual  information  for  a  limited  number  of  data  fields.  The  VCE-BI  module  has  the 
ability  to  pull  data  from  FPDS-NG  and  the  VCE- Acquisition  Management  module 
(which  contains  detailed  contract  data  beyond  the  infonnation  captured  in  the  Contract 
Action  Report  [CAR]  for  FPDS-NG). 

(2)  Federal  Procurement  Data  System-Next  Generation 

The  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG)  contains 
contractual  data  for  contract  actions  with  estimated  values  of  $3,000  or  more,  or  that  may 
be  $3,000  or  more,  including  every  modification  to  that  contract  (“About  FPDS-NG,” 
n.d.).  The  contract  data  captured  on  a  CAR,  which  is  completed  by  the  Contracting 
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Officer  after  each  contract  action,  is  stored  in  FPDS-NG,  and  fed  to  other  outside 
systems,  including  VCE. 

(3)  Paperless  Contract  Files 

The  Paperless  Contract  Files  (PCF)  system  is  a  module  within  VCE  that  is  a 
complete  document  management,  storage  and  workflow  solution  for  the  contracting 
workforce  at  ACC.  PCF  was  developed  by  ACC  to  assist  the  contracting  workforce  to 
manage  immense  amounts  of  documents  contained  in  a  traditional  contract  file  in  one 
place.  PCF  is  a  secure,  web-based  application  that  can  be  accessed  by  the  contracting 
workforce  at  ACC,  and  archives  historical  files  that  were  originated  in  the  system  by  a 
member  of  ACC.  Electronic  contract  file  documents  were  retrieved  from  PCF  to  create  a 
digital  contract  file  that  was  reviewed  to  obtain  the  data  for  this  research. 

(4)  Contractor  Performance  Assessment  Reporting  System 

CPARS  is  a  web-enabled  system  designed  to  host  a  history  of  contractor 
perfonnance  on  specific  contracts,  as  submitted  by  the  receiver  of  the  contractor’s 
supplies  or  services.  CPARS  is  a  system  of  processes  that  assists  the  assessing  official  in 
assembling  a  review  of  a  contractor’s  performance  based  on  various  limited  elements, 
including  quality  of  the  product/service,  schedule  quality,  ability  to  control  cost,  quality 
of  business  relations,  utilization  of  small  business,  and  key  personnel  management,  as 
relevant  to  the  specific  contract.  The  primary  purpose  of  the  CPARS  is  to  ensure  that 
current,  complete  and  accurate  information  on  contractor  performance  is  available  for  use 
in  procurement  source  selections  (Guidance,  2014,  p.  13). 

According  to  the  CPARS  Guide,  and  FAR  Part  42.15,  CPARS  reports  are 
required  for  any  action  in  excess  of  the  Simplified  Acquisition  Threshold,  and  over  $1M 
for  Information  Technology  (see  Figure  2)  (Guidance,  2014,  p.  21).  CPARS  is  meant  to 
capture  data  for  unclassified  contracts  only. 
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Business  Sector 

Dollar  Threshold1 

Reviewing  Official2 

Civilian  .Agencies  (excludes  DoD): 

Systems  and  Xon-Systems 

>  Simplified  Acquisition 
Threshold 

One  level  above  the  Contracting 
Officer,  as  determined  by 
Department  or  Agency  policy 

Architect-Engineer 

>$30,000: 

All  Terminations  for 
Default 

One  level  above  the  Contracting 
Officer,  as  determined  by 
Department  or  Agency  policy 

Construction 

>$650,000; 

All  Terminations  for 
Default 

One  level  abov  e  the  Contracting 
Officer,  as  determined  by 
Department  or  Agency  policy 

DoD  Services  and  Agencies: 

Systems  (includes  new  development  and 
major  modifications) 

>$5,000,000 

One  level  abov  e  the  PM1 

N’on-Systems 

Operations  Support 

>$5,000,000* 

One  level  above  the  AO 

Services 

>$1,000,000 

One  level  above  the  AO 

Information  Technology 

>$1,000,000 

One  level  above  the  AO 

Ship  Repair  and  Overhaul 

>$500,000 

One  level  above  the  AO 

Architect-Engineer 

>$30,000: 

All  Terminations  for 
Default 

One  lev  el  abov  e  the  AO 

Construction 

>$650,000; 

All  Terminations  for 
Default 

One  level  above  the  AO 

'The  contract  order  thresholds  for  CPAR  collection  (see  FAR  42.1502)  apply  to  the  "  aggregate"  value  of 
contracts  orders;  that  is,  if  a  contract  Vorder  s  original  award  value  were  less  than  the  applicable  threshold  but 
subsequently  the  contract  order  was  modified  and  the  new  value  is  greater  than  the  threshold,  then  evaluations  are 
required  to  be  made,  stalling  with  the  first  anniversary  that  the  contract’s/order  s  face  value  exceeded  the  thieshold. 
If  the  total  contract  order  value  including  unexercised  options  and  orders  (for  IDIQ  contracts,  total  estimated  value 
of  unexercised  options  and  orders)  is  expected  to  exceed  the  collecnon  threshold,  inmate  the  collection  process  at 
the  start  of  the  contract ’order.  Buying  activities  may  choose  to  collect  performance  evaluattons  for  awards  below 
these  thresholds . 

2Only  required  when  the  contractor  indicates  non -concurrence  with  the  CPAR  or  if  otherwise 
requested  by  the  contractor  during  the  60-calendar  day  comment  period. 

J(Or  equivalent  individual)  responsible  for  program,  project  or  task'job  order  execution 

*For  contracts  orders  under  the  reporting  thresholds,  buying  activities  should  continue  to  accumulate  contactor 
performance  data  from  existing  management  information  systems,  which  already  capture  data  on  timeliness  of 
delivery'  and  quality  of  product  or  service  (an  example  of  a  performance  informaton  collecnon  systems  is  the  Past 
Performance  Infoimanon  Retrieval  System  -  Stattstcal  Reporting  Module 


Figure  2.  Business  Sector,  Dollar  Threshold  and  Reviewing  Official 

(from  CPARS,  2014) 
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Dollar  thresholds  are  applied  to  the  aggregate  value  of  the  contract  action; 
therefore  if  a  contract  is  modified  over  time  to  increase  the  total  dollar  value,  then  if  the 
new  value  exceeds  the  threshold,  a  CPARS  report  is  required.  If  a  contract  were  to 
remain  below  that  threshold,  however,  the  CPARS  report  is  not  required.  By  only 
requiring  past  performance  infonnation  for  larger  dollar  procurements,  countless  smaller 
dollar  procurements  go  unevaluated.  Therefore,  this  study  reviews  CPARS  reports 
specifically  for  contracts  related  to  IT  with  contract  values  greater  than  $1M  for  PEO  EIS 
in  order  to  have  a  higher  likelihood  of  obtaining  a  complete  CPARS  report. 

C.  COLLECTION  OF  CONTRACT  DATA 

This  section  will  discuss  the  criteria  utilized  to  assemble  a  list  of  eligible  contracts 
for  this  study,  as  well  as  the  data  collected  from  those  contracts. 

1.  Criteria  for  VCE  Report 

Ad-hoc  reporting,  in  terms  of  business  intelligence  reports,  refers  to  a  report  that 
is  assembled  “as  the  occasion  requires”  based  on  the  needs  of  the  end-user.  For  this  study 
a  report  was  generated  with  the  following  criteria: 

•  C-Type  Contracts  (later  re-ran  the  report  to  include  all  types  of  contracts, 
as  PEO  EIS  has  fewer  C-type  contracts) 

•  Closed  between  2001  and  2014  (for  greater  likelihood  for  completed 
CPARS  reports) 

•  Above  the  SAT 

•  Best  value  method  listed  (LPTA  or  tradeoff)  (later  re-ran  the  report  to 
disregard  this  field,  as  the  VCE-Acquisition  Management  module  was  not 
required  until  2012,  indicating  that  this  field  does  not  capture  100%  of 
contract  files  for  the  specified  time  range) 

This  report  was  provided  by  the  VCE  data  helpdesk  based  on  PEO  EIS  contract 
data  that  was  entered  into  FPDS-NG  from  CAR  reports  created  by  contracting  officers 
for  the  respective  contract  actions.  The  resulting  report  of  124  contract  actions  were 
deemed  acceptable  potential  files  for  this  project.  The  list  of  contracts  was  sent  to  the 
leadership  at  ACC-Rock  Island  Contracting  Center  at  the  Rock  Island  Arsenal  in  Illinois 
and  to  the  ACC-New  Jersey  Contracting  Center  at  Picatinny  Arsenal  in  New  Jersey,  as 
these  two  contracting  centers  facilitate  in  the  majority  of  PEO  EIS  major  contract  actions. 
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The  files  that  were  retrievable  from  the  electronic  systems  were  provided  and  evaluated 
for  information  in  accordance  with  the  aforementioned  data  fields. 


2.  Data  Collection  Rubric 

The  data  collection  rubric  was  designed  to  capture  96  potential  data  points  that  are 
typically  found  in  complex  contract  actions.  The  five  categories  of  data  collected 
included: 

•  Basic  Contract  Infonnation 

•  Acquisition  Complexity 

•  Environmental  Factors 

•  Outcome  Variables 

•  Other 

Basic  Contract  Information  included  the  contract  number,  name  of  the  project, 
North  American  Industry  Classification  System  (NAICS)  code  and  Product  Service  Code 
(PSC). 

Acquisition  Complexity  measured  the  complexity  of  the  procurement  with  fields 
such  as  the  use  of  LPTA  or  tradeoff,  number  and  type  of  evaluation  factors,  the  FAR 
section  employed  in  the  acquisition  (FAR  Part  12,  13  or  15),  use  of  incentives,  etc.  This 
data  created  a  broad  understanding  of  the  difficulty  of  the  acquisition,  the  technicality  of 
forethought  on  the  part  of  the  acquisition  team,  and  lead  into  evaluating  the 
environmental  factors. 

Environmental  Factors  took  measure  of  the  circumstances  that  surrounded  the 
source  selection,  such  as  number  of  reviews  of  the  proposals,  number  of  personnel  on  the 
source  selection  team,  the  quality  of  the  file  documentation.  This  data  created  a  broad 
understanding  of  the  involvedness  of  the  acquisition  team  in  the  procurement  and  the 
difficulties  that  may  have  occurred  in  selecting  a  vendor. 

Outcome  Variables  included  the  measurements  pulled  from  the  CPARS  reports 
(Cost,  Quality,  etc.),  Earned  Value  Management  (EVM),  where  applicable,  whether  or 
not  a  protest  took  place,  and  other  such  measurements  of  contract  outcomes.  This  data 
was  important  to  connect  to  the  complexity  of  the  acquisition  to  find  a  potential  cause- 

and-effect  relationship  between  the  complexity,  the  method,  and  the  outcomes  of  either 
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LPTA  or  tradeoff.  Other  data  fields  captured  included  number  of  Evaluation  Notices, 
Clarification  Requests,  and  Deficiency  Reports. 


D.  DATA  DESCRIPTION 

This  study  is  aiming  to  establish  whether  the  CPARS  ratings  are  impacted  by  the 
choice  of  LPTA  or  tradeoff  methods  during  source  selection  evaluation. 

The  Dependent  Variable  (DV)  is  the  outcome  or  effect  of  a  change  in  the 
Independent  Variable  (IV),  the  input  or  cause  in  the  equation.  For  this  analysis  the 
CPARS  rating  and  the  PALT  are  the  DVs.  PALT  is  the  calculation  of  the  time  leading  up 
to  award  (in  days),  starting  at  the  time  the  requirements  package  is  received  by  the 
contracting  office.  The  CPARS  ratings  are  a  proxy  measure  of  contract  success  or  failure, 
based  upon  the  reviews  provided  by  the  customer.  CPARS  ratings  are  provided  by  the 
assessing  official  in  the  CPARS  report  cards  for  areas  such  as  Cost,  Quality  and 
Schedule,  ranging  from  Unsatisfactory  (not  meeting  the  standard  contractual 
expectations)  to  Excellent  (exceeding  contractual  expectations).  The  ratings  are  given  in 
Likert-style  responses  where  l=Unsatisfactory,  2=Marginal,  3=Satisfactory,  4=Very 
Good,  and  5=Excellent.  For  this  study  the  overall  CPARS  score  was  calculated  by 
averaging  the  following  CPARS  factors:  quality,  schedule,  management  of  key 
personnel,  and  small  business  use.  The  average  score  was  used  because  of  the  limited 
sample  size;  for  future  research  with  larger  sample  sizes  each  CPARS  rating  could  be 
used  individually  to  measure  contract  success  or  failure  in  each  rating  category.  In  this 
case,  both  individual  and  combined  CPARS  scores  were  used  (note:  individual  scores 
were  examined  without  success  due  to  data  limitations).  Specifically,  differences  in 
CPARS  rating  between  LPTA  and  tradeoff  acquisition  strategies  were  sought  by 
examining  CPARS  quality  ratings,  cost  ratings,  and  CPARS  business  relationship  rating 
individually,  and  also  by  looking  for  differences  using  the  average  CPARS  rating  for 
each  acquisition  (in  other  words,  the  average  of  quality,  cost  and  business  relationship 
scores). 

The  independent  variable  (IV)  is  the  choice  of  utilizing  either  LPTA  or  tradeoff 
method,  as  this  variable  is  detennined  by  the  acquisition  team  and  therefore  could  impact 
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the  DV.  The  acquisition  team  chooses  LPTA  or  tradeoff  prior  to  RFP  release  and  source 
selection  evaluation.  The  IV  is  labeled  LPTATO  and  it  is  a  binary  variable  where 
0=LPTA  and  l=tradeoff. 

Finally,  there  is  one  covariate  variable.  Covariates  are  secondary  variables  that 
can  also  affect  the  relationship  of  primary  interest:  the  relationship  between  the  IV  and 
the  DV.  Specifically,  covariates  are  variables  other  than  the  independent  variable  that 
potentially  affect  the  outcome  variable,  or  DV.  In  this  case,  the  covariate  is  contract 
dollar  value  (VALUE),  and  it  is  a  continuous  variable.  The  dollar  value  of  a  contract 
affects  the  number  of  reviews  it  has  to  go  through,  thus  affecting  the  PALT.  Higher  dollar 
contracts  typically  endure  more  reviews,  and  thus  have  longer  PALTs.  The  opposite  is 
typically  true  for  lower  dollar  value  contracts.  In  this  case,  the  goal  is  to  parcel  out  the 
effect  of  the  covariate  VALUE  in  order  to  more  clearly  see  the  effect  the  contracting 
methodology  (LPTATO)  has  on  the  outcome  variables  (PALT  and  CPARS  ratings). 

The  goal  of  this  analysis  is  to  see  if  there  is  a  cause-effect  relationship  between 
the  choice  of  evaluation  method  and  the  PALT  and  CPARS  ratings.  This  analysis  will 
measure  whether  there  is  a  relationship  between  the  DVs  and  the  IV,  and  the  degree  of 
strength  or  significance  of  that  relationship. 

E.  SUMMARY 

This  chapter  has  reviewed  the  methodology  for  this  study,  including  a  brief 
overview  of  the  contracting  systems  used  to  gather  reports  and  data,  how  the  contract 
data  was  acquired,  description  of  the  data,  descriptive  statistics,  and  data  issues.  The  next 
chapter  discusses  the  data  analysis  and  findings. 
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IV.  DATA  ANALYSIS  AND  FINDINGS 


This  chapter  explains  the  data  analysis  and  statistical  findings  that  resulted  from 
the  assessment.  Explanation  of  the  statistical  methods  used,  assumptions  applied  to 
normalize  the  data,  analysis,  and  whether  or  not  the  research  questions  were  answered  are 
discussed. 

A.  DESCRIPTIVE  STATISTICS 

Basic  descriptive  statistics  for  each  variable  are  shown  in  Table  1.  The  table 
presents  three  figures  for  each  variable:  (1)  the  total  for  all  the  data,  (2)  the  total  for 
LPTA  contracts,  and  (3)  the  total  for  tradeoff  contracts. 
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] 

Descriptive  Statistics 

Variable 

Obs 

Mean 

SD 

Min 

Max 

PALT 

(days) 

8 

256.88 

392.16 

50 

1216 

ALL  DATA 
TOTAL 

3 

156.33 

94.56 

50 

231 

LPTA 

5 

317.20 

502.52 

84 

1216 

TRADEOFF 

CPARS 

Quality 

(rating) 

19 

3.47 

.96 

1 

5 

ALL  DATA 
TOTAL 

14 

3.50 

1.02 

1 

5 

LPTA 

5 

3.40 

.89 

3 

5 

TRADEOFF 

CPARS 

Cost 

(rating) 

9 

3.56 

1.13 

1 

5 

ALL  DATA 
TOTAL 

8 

3.88 

.64 

3 

5 

LPTA 

1 

1 

— 

1 

1 

TRADEOFF 

CPARS 

Business 

Relationship 

(rating) 

19 

3.42 

.96 

1 

5 

ALL  DATA 
TOTAL 

14 

3.43 

1.02 

1 

5 

LPTA 

5 

3.40 

.89 

3 

5 

TRADEOFF 

CPARS 

Overall 

Average 

(rating) 

19 

3.42 

.99 

1 

5 

ALL  DATA 
TOTAL 

14 

3.45 

1.01 

1 

5 

LPTA 

5 

3.27 

1.01 

2.33 

5 

TRADEOFF 

VALUE 

(dollars) 

19 

$3,536,880 

$4,263,114 

$13,370 

$17,800,000 

ALL  DATA 
TOTAL 

14 

$2,639,221 

$2,851,057 

$13,370 

$7,700,654 

LPTA 

5 

$6,050,324 

$6,681,982 

$1,613,496 

$17,800,000 

TRADEOFF 

Table  1.  Descriptive  Statistics  for  Data  Set 


B.  DATA  ISSUES 

The  report  that  was  retrieved  from  VCE  contained  a  number  of  contract  files  that 
were  originated  at  the  Anny  Contracting  Command,  National  Capital  Region  (ACC- 
NCR),  which  cased  its  colors  and  closed  on  July  11,  2013  (Gore,  2013).  The  contract 
files  that  were  originated  at  ACC-NCR  for  PEO  EIS  were  not  fully  accessible  as  those 
files  were  not  consistently  stored  in  the  PCF  module  prior  to  the  closure  of  the 
contracting  center.  Therefore,  a  significant  portion  of  data  that  may  have  assisted  in  this 
research  was  not  accessible.  This  project,  therefore,  is  an  incomplete  analysis  of 
acquisition  history  for  PEO  EIS,  given  the  recent  migration  to  utilizing  a  new  group  of 
contracting  centers  since  2013.  This  limitation  on  data  collection  should  be  considered 
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throughout  the  study;  future  research  could  benefit  from  gathering  contractual  data  for 
longer  than  two  years  from  a  relatively  new  contracting  organization. 

The  documentation  for  the  files  that  were  electronically  accessible  was  sparse, 
indicating  that  multiple  contracting  professionals  had  worked  the  actions  without  a  single 
point  responsible  for  complete  contract  file  management,  resulting  in  relatively 
incomplete  electronic  contract  files.  The  factor  of  human  error  should  be  considered  here, 
as  internal  controls  are  not  wholly  monitoring  the  management  of  complete  contract  files 
within  the  system.  Internal  controls  for  historical  data  filing  are  another  issue  for  potential 
future  research. 

With  only  19  cases,  the  sample  size  is  small.  Power  calculations  suggest  the  need 
for  14  cases  for  each  contracting  methodology  (i.e.,  14  LPTA  cases  and  14  tradeoff 
cases)  in  order  to  achieve  adequate  power  (a  =  .05,  P  =  .80).  The  data  are  unbalanced 
with  respect  to  the  number  of  cases  for  each  contracting  methodology.  There  are  14 
LPTA  cases  and  5  tradeoff  cases.  This  unbalanced  design  can  cause  ambiguity  about  the 
mean  as  the  intercept  and  make  assignment  of  sums  of  squares  more  difficult.  There  are, 
however,  solutions  to  these  issues.  A  weighted  mean  can  be  used  in  place  of  the  grand 
mean  (normally  the  intercept  in  a  balanced  design),  and  the  Stata  software  (version  12.1) 
automatically  handles  the  assignment  of  the  sums  of  squares.  Thus,  we  proceeded  with 
our  analysis  despite  these  issues. 

C.  DATA  ANALYSIS 

Because  the  intent  is  to  analyze  differences  in  contract  outcomes  (PALT  and 
CPARS  ratings)  based  on  contracting  methodology  (LPTA  or  tradeoff),  a  group 
comparison  statistical  methodology  is  necessary.  In  other  words,  the  contracting 
methodologies  are  divided  into  two  groups  (LPTA  and  tradeoff),  and  analyzed  to  find  if 
there  are  differences  in  contract  outcomes  (PALT  and  CPARS  ratings)  by  group. 

A  technique  called  multivariate  analysis  of  covariance  (MANCOVA)  was  initially 
used  to  assess  group  differences.  Results  could  not  be  obtained  given  the  high  collinearity 
between  the  variables.  In  the  case  of  MANCOVA,  collinearity  occurs  when  the 
dependent  variables  are  very  highly  correlated  with  each  other,  to  the  point  of  being 
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indistinguishable  from  each  other.  The  general  rule  of  thumb  is  that  dependent  variables 
correlated  at  .80  or  higher  are  likely  to  be  collinear  (Dattalo,  2013).  In  this  case,  all  of  the 
dependent  variables  are  correlated  at  .78  or  higher,  preventing  accurate  analysis. 

Given  the  existing  collinearity,  each  of  the  dependent  variables  was  examined 
separately.  In  other  words,  differences  in  contract  outcomes  (PALT  and  CPARS  ratings) 
were  sought  out  individually.  Although  multivariate  methods  like  MANCOVA  paint  a 
more  complete  picture  of  the  relationships  between  contracting  strategy  (LPTA  or 
tradeoff)  and  subsequent  outcomes,  univariate  techniques  can  also  help  piece  that  picture 
together,  albeit  in  a  fragmented  way.  Thus  post-hoc  analyses  were  performed  in  which 
PALT  and  CPARS  ratings  were  analyzed  separately.  Again,  because  the  outcome 
variables  were  assessed  individually,  the  methodology  changed  from  a  multivariate  test 
(MANCOVA)  to  a  univariate  test,  known  simply  as  analysis  of  covariance  (ANCOVA). 

ANCOVA  addresses  the  following  questions:  Are  mean  differences  among  the 
groups  (after  adjusting  for  covariate  effects)  likely  to  have  occurred  by  chance?  Taken 
from  another  angle,  is  there  a  significant  difference  between  the  mean  value  for  DVs 
(PALT  and  CPARS  ratings)  in  the  LPTA  acquisitions  versus  the  mean  value  for  DVs 
(PALT  and  CPARS  ratings)  in  the  tradeoff  acquisitions  once  the  effect  of  the  covariate 
(contract  dollar  value,  referred  to  as  VALUE)  has  been  parceled  out?  ANCOVA 
examines  the  relationships  between  the  dependent  variables  (PALT  and  CPARS  ratings) 
and  the  independent  variable  (choice  of  LPTA  or  tradeoff  methodology)  while  taking  into 
account  the  effect  the  covariate  (VALUE)  might  have  on  the  outcome  variables  (PALT 
and  CPARS  ratings). 

D.  ASSUMPTION  TESTING 

Before  conducting  the  ANCOVA,  certain  assumptions  about  the  data  were  tested. 
First,  univariate  normality  was  assessed  by  performing  the  Shapiro-Wilk  test  to  check 
whether  a  sample  came  from  a  normally  distributed  population.  Both  PALT  and  contract 
value  (VALUE)  were  deemed  to  be  non-nonnal.  Both  variables  were  nonnalized  via  a 
logarithmic  transformation.  Both  variables  passed  the  Shapiro-Wilk  nonnality  test  once 
transfonned. 
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Second,  linearity  was  assessed  by  examining  scatter  plots  of  the  dependent 
variables  (PALT  and  CPARS  ratings)  and  the  covariate  variable  (VALUE).  The  plots 
revealed  linear  relationships  between  the  variables. 

Third,  homogeneity  of  regression  was  assessed  by  perfonning  an  analysis  of 
variance  (ANOVA,  which  is  similar  to  an  ANCOVA  but  does  not  contain  a  covariate 
variable)  that  included  the  independent  variable  (LPTA  or  tradeoff),  the  covariate 
VALUE,  and  the  interaction  between  the  independent  variable  and  the  covariate.  The 
interaction  term  was  not  significant,  which  indicates  that  the  relationship  between  the 
dependent  variables  (PALT  and  CPARS  ratings)  and  covariate  (VALUE)  is  the  same  at 
both  levels  of  the  independent  variable  (LPTA  or  tradeoff).  Hence,  the  assumption  of 
homogeneity  of  regression  is  upheld. 

Finally,  homogeneity  of  variance  between  groups  was  assessed  using  Bartlett’s 
Test.  For  most  of  the  outcome  variables,  the  results  showed  the  differences  in  variance 
among  the  groups  (LPTA  and  tradeoff)  are  not  significant,  thus  the  assumption  of 
homogeneity  of  variance  is  upheld.  The  contract’s  Cost  rating  (i.e.,  the  CPARS  rating 
that  assesses  cost  performance  of  the  contractor  using  a  Likert-type  scale)  only  had  one 
case  for  the  tradeoff  methodology  (8  cases  for  the  LPTA  methodology),  thus 
homogeneity  of  variance  could  not  be  adequately  assessed  for  that  outcome  variable. 

E.  RESULTS 

The  results  of  the  analysis  show  that  the  only  situation  where  source  selection 
methodology  (LPTA  or  tradeoff)  produced  significantly  different  contracting  outcomes 
was  in  the  case  of  the  contractor’s  CPARS  cost  rating.  Essentially,  the  cost  rating  is  lower 
for  tradeoff  contracts.  However,  because  there  was  only  one  tradeoff  case,  these  results 
cannot  be  relied  on  with  any  firm  accuracy.  More  cases  are  needed  to  confirm  these 
results,  as  will  be  addressed  in  Chapter  V  regarding  areas  for  further  research.  Table  2 
shows  the  results  of  this  analysis. 
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Source 

Partial  SSA 

df 

MSAA 

F 

Prob  >  F 

Model 

7.57 

2 

3.79 

8.57 

.0174* 

LogVALUE 

.23 

1 

.23 

.51 

.5020  ns 

LPTATO 

6.39 

1 

6.39 

14.48 

.0089** 

Residual 

2.65 

6 

.44 

Total 

10.22 

8 

1.28 

* p<. 05  **p<.01 

A  Partial  sum  of  squares  AA  Mean  square 
Number  of  Observations  =  9 

Root  Mean  Squared  Error  =  .66 

R2  = . 7408 

Adjusted  R2  =  .6544 

Table  2.  Analysis  of  the  Covariance  Using  CPARS  Cost  Rating  as  the  DV 


F.  ADDITIONAL  ANALYSIS 

Perhaps  more  telling  from  this  sample  are  the  relationships  between  the  variables 
themselves.  The  correlations  are  shown  in  Table  3. 


PALT 

CPARS 

Avg 

CPARS 

Quality 

CPARS 

Cost 

CPARS 

Business 

Relationship 

LPTATO 

VALUE 

PALT 

1.00 

CPARS 

Avg 

-0.78 

1.00 

CPARS 

Quality 

-0.94 

0.94 

1.00 

CPARS 

Cost 

-0.78 

1.00 

0.94 

1.00 

CPARS 
Bus  Rel 

-0.88 

0.98 

0.99 

0.98 

1.00 

LPTATOA 

0.99 

-0.82 

-0.96 

-0.82 

-0.90 

1.00 

VALUE 

0.95 

-0.87 

-0.95 

-0.87 

-0.92 

0.93 

1.00 

A Point  biserial  correlation  because  LPTATO  is  a  binary  variable  where 

0  =  LPTA  and  1  -  tradeoff 

Table  3.  Correlation  Table — All  Data 
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The  items  in  the  table  with  a  correlation  coefficient  +0.70  between  the  variable  in 
the  first  column  and  the  variables  in  the  column  headers  indicates  a  strong  positive  linear 
relationship  between  those  two  variables.  There  appears  to  be  a  strong  positive  linear 
relationship  between  LPTATO  and  PALT  (.99)  and  contract  value  and  PALT  (.95).  This 
relationship  is  most  likely  due  to  the  notion  that  complex,  higher  dollar  contracts  take  a 
longer  time  to  award;  therefore,  tradeoff  acquisitions  (LPTATO  =1)  take  more  time  to 
award  (an  increased  PALT  time). 

The  items  in  the  table  with  a  correlation  coefficient  -0.70  between  the  variable  in 
the  first  column  and  the  variables  in  the  column  headers  indicates  a  strong  negative  linear 
relationship  between  those  two  variables.  There  appears  to  be  a  strong  negative  linear 
relationship  between  the  CPARS  ratings  and  PALT  (valuing  from  -.78  to  -.88  to  -.94  for 
Cost,  Business  Relationship  and  Quality,  respectively,  with  the  correlation  between  the 
average  CPARS  scores  and  PALT  being  -.78),  suggesting  that  the  CPARS  ratings  decline 
as  the  PALT  (days)  increases.  This  relationship  might  indicate  that  even  with  increased 
time  dedicated  to  a  procurement,  there  is  no  guarantee  that  the  CPARS  ratings  will 
improve.  PALT  can  be  increased  due  to  bureaucratic  levels  of  process  reviews  that  do  not 
add  value,  therefore  PALT  is  not  always  an  indicator  of  increased  quality  in  the  product 
(the  acquisition  package).  A  low-quality  acquisition  package  may  lead  to  confusion 
between  the  government  and  the  contractor  regarding  deliverables,  thereby  resulting  in 
lower  CPARS  ratings.  Likewise,  the  shorter  PALT  could  be  the  result  of  simple  LPTA 
acquisitions  (also  known  for  low  complexity,  clear  requirements),  therefore  positive 
CPARS  rating  are  easier  to  achieve.  An  increase  in  data  could  lend  more  insight  to  this 
relationship  and  confirm  these  notions. 

After  the  original  correlations  were  analyzed,  analysis  was  performed  on  the 
relationships  between  the  LPTA  acquisitions  and  the  CPARS  outcomes  (see  results  in 
Table  4). 
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PALT 

CPARS 

Quality 

CPARS 

Cost 

CPARS 

Business 

Relationships 

CPARS 

Value 

PALT 

1 

CPARS 

Quality 

0.29 

1 

CPARS  Cost 

0.29 

1 

1 

CPARS 

Business 

Relationships 

0.29 

1 

1 

1 

CPARS 

0.29 

1 

1 

1 

1 

Value 

0.6555 

-0.5326 

-0.5326 

-0.5326 

-0.5326 

1 

Table  4.  Correlations  When  LPTA  Methodology  Was  Chosen 


The  data  in  Table  4  shows  that  the  relationship  between  and  PALT  and  CPARS 
for  LPTA  acquisitions  is  0.29,  indicating  a  moderately  positive  linear  relationship.  This 
indicates  that  for  acquisitions  with  a  longer  PALT  time,  the  CPARS  rating  may  be  better, 
possibly  the  result  of  better  efforts  being  applied  to  the  contract  setup  leading  to  better 
contractor  performance  overall.  There  is  a  negative  relationship  between  the  contract 
value  and  the  CPARS  ratings  for  the  LPTA  acquisitions,  suggesting  that  higher  dollar 
LPTA  contracts  are  not  viewed  to  be  as  successful  as  lower  dollar  LPTA  contracts  (in 
terms  of  CPARS  ratings). 

The  relationship  between  PALT  and  contract  value  for  LPTA  acquisitions  is  a 
stronger  positive  relationship  at  0.65,  perhaps  a  result  of  the  complexity  of  the  contract  - 
the  higher  the  dollar  value  on  the  contract,  the  more  complex  the  contract,  the  longer  the 
PALT. 

Analysis  was  also  performed  on  the  relationships  between  the  tradeoff 
acquisitions  and  the  CPARS  outcomes  (see  results  in  Table  5).  In  Table  5,  the  CPARS 
Cost  rating  was  removed  because  there  was  only  one  value,  resulting  in  skewed  statistics 
due  to  insufficient  observations. 
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PALT 

CPARS 

Quality 

CPARS 

Business 

Relationships 

CPARS 

Cost 

Value 

PALT 

1 

CPARS 

Quality 

-0.2594 

1 

CPARS 

Business 

Relationships 

-0.2594 

1 

1 

CPARS 

-0.5243 

0.9584 

0.9584 

1 

Value 

0.9836 

-0.3712 

-0.3712 

-0.6181 

1 

Table  5.  Correlations  When  Tradeoff  Methodology  Was  Chosen 


As  discussed  previously,  the  relationship  between  PALT  and  contract  value  for 
tradeoff  acquisitions  is  a  stronger  positive  relationship  at  0.9836,  almost  a  perfect 
relationship.  This  may  be,  again,  a  result  of  the  complexity  of  the  contract. 

There  is  a  negative  linear  relationship  between  the  PALT  and  the  CPARS  ratings 
for  tradeoff  acquisitions,  at  -0.2594,  indicating  that  as  the  PALT  increases  the  CPARS 
ratings  for  quality  and  business  relationships  decrease.  There  could  be  multiple  causes  for 
this  pattern.  A  longer  PALT  is  indicative  of  a  potentially  lengthy  resultant  contract.  A 
long  contract  term  with  negative  ratings  could  be  indicative  of  an  increasingly  stressful 
work  environment,  or  drawn-out  contentious  disputing  over  deliverables.  Longer  contract 
terms  offer  more  chances  for  the  government  COR  to  witness  positive,  but  also  negative, 
aspects  of  the  contractor’s  perfonnance,  leading  to  negative  performance  reviews.  The 
CPARS  reports  would  need  to  be  reviewed  in  depth  to  find  answers  for  this  relationship. 

G.  SUMMARY 

This  section  discussed  the  data  analysis  and  statistical  findings  that  resulted  from 
this  study,  and  the  statistical  methods  and  assumptions  employed  to  calculate  and  these 
results.  A  number  of  post-hoc  analyses  were  employed  to  further  understand  the  data. 
These  additional  analyses  revealed  positive  and  negative  correlational  relationships 
between  covariate  and  outcome  variables.  The  next  and  final  chapter  of  this  study 
discusses  the  conclusions  and  recommendations  based  on  these  findings. 
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V.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 


The  purpose  of  this  chapter  is  to  summarize  the  concepts  and  findings  from  this 
study.  This  chapter  will  discuss  the  managerial  implications  of  the  results  found  in 
Chapter  IV,  answer  the  research  questions  posited  at  the  beginning  of  this  study,  and 
make  recommendations  for  future  research.  This  study  only  scratches  the  surface  of  best 
value  application  in  the  Anny,  and  has  ultimately  raised  more  questions  for  future 
research. 

A.  SUMMARY 

The  goal  of  this  project  was  to  detennine  whether  or  not  there  is  a  relationship  in 
U.S.  Army  contracts  between  the  method  of  procurement  (LPTA  or  tradeoff)  and  the 
quality  of  the  contract  outcomes,  in  this  case  measured  by  evaluation  of  CPARS  reports 
and  PALT.  This  goal  was  met,  to  a  degree,  given  the  challenges  of  limited  data  and 
influence  of  other  factors  on  statistical  significance.  Further  experimentation  is  necessary 
to  detennine  if  the  results  of  this  experiment  are  applicable  across  the  Anny  and  the 
DOD. 

Source  selection  methodologies  along  the  best  value  continuum  are  used 
extensively  in  defense  contracting,  but  the  extent  to  which  acquisition  personnel 
understand  how  to  appropriately  use  best  value  may  be  limited.  While  GAO  reports  show 
that  most  high-cost  defense  acquisitions  are  utilizing  best  value,  there  is  indication  that 
non-cost  evaluation  factors  are  not  being  used  to  their  full  potential.  If  requirements  are 
well-written  and  there  is  low  risk  for  contract  performance,  then  LPTA  is  often  the  best 
process.  If  the  government  has  prioritized  non-cost  factors  to  be  equal  to  or  greater  than 
the  cost  in  terms  of  important,  or  is  seeking  innovative  solutions,  then  tradeoff  is  typically 
the  best  process.  Non-cost  factors  can  be  monetized  when  necessary,  and  by  doing  so  the 
risk  for  a  protest  based  on  subjectivity  can  be  mitigated.  By  monetizing  even  non-cost 
factors  there  is  room  to  justify  an  acquisition  decision,  thereby  putting  fears  of  protest  at 
ease.  The  government  acquisition  team  must  be  honest  and  thorough  when  developing 
the  requirements  and  priorities  of  an  acquisition.  Determining  priorities  of  cost  and  non- 
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cost  factors  on  the  best  value  continuum  are  the  lynchpin  to  a  successful  evaluation.  All 
parties  should  be  cognizant  of  the  pressures  to  make  a  timely  award,  reducing  costs, 
reducing  risks,  limited  resources,  as  all  are  factors  that  can  very  much  inhibit  a  successful 
evaluation  and  make  all  efforts  to  disallow  their  encumbrance  on  the  evaluation. 

Government  and  industry  have  conflicting  perceptions  of  best  value,  how  it  is 
used  and  what  it  offers.  While  government  views  best  value  as  a  means  to  get  the  most 
for  a  dollar,  industry  realizes  that  best  value,  when  improperly  used,  often  results  in 
conflicting  information  during  negotiations,  “low-cost,  low-quality,”  or  instances  in 
which  cost  becomes  a  “negligible”  factor  in  evaluations  (Biddle,  1993,  p.  128).  The  DOD 
has  made  strides  with  Better  Buying  Power  and  other  efforts  to  reform  best  value 
contracting,  but  industry  continues  to  find  areas  for  improvement  in  the  contract 
management  process.  While  efforts  have  been  made  to  bridge  this  conflict,  all  can  agree 
that  if  the  principles  of  best  value  are  improperly  applied  that  the  results  can  be  disastrous 
for  private  industry  and  for  the  taxpayer. 

There  is  a  lingering  need  for  acquisition  education  in  the  DOD  workforce.  DAU 
has  made  marked  improvement  on  acquisition  education,  but  theoretical  information  is 
only  so  helpful;  on-the-job  training  is  valued  by  many  to  be  the  most  valuable  source  of 
obtaining  acquisition  education,  particularly  for  source  selections.  With  the  increasing 
defense  mission  and  dwindling  resources  of  time,  money  and  people,  however,  there  just 
is  not  enough  guidance  to  assist  the  workforce  in  achieving  source  selection 
enlightenment.  The  Better  Buying  Power  mantra  of  “do  more  without  more”  has  been 
pushed  beyond  doing  more  with  fewer  dollars  into  perfonning  more  acquisitions  with 
fewer  manpower  resources.  The  lack  of  resources  is  also  resulting  in  a  shift  to  using 
LPTA  more  often  than  tradeoff  purely  because  LPTA  requires  fewer  resources  to 
accomplish  the  same  mission.  As  more  stress  is  put  on  the  acquisition  process,  there  is 
higher  risk  for  making  poor  business  decisions.  If  the  best  value  processes  are  used 
appropriately  and  with  proper  intent,  the  government  has  the  highest  chance  to  receive 
the  best  value  for  the  taxpayer  dollars.  This  study  therefore  aims  to  determine  whether 
there  is  a  likelihood  for  better  contract  perfonnance  outcome  using  a  particular  evaluation 
process.  The  goal  is  to  provide  the  acquisition  workforce  with  better  expectations  for 
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post-award  performance,  so  that  it  may  be  considered  when  choosing  the  most 
appropriate  source  selection  process. 

Post-award  evaluation  to  build  past  perfonnance  is  the  often  forgotten  but 
increasing  crucial  phase  of  contracting.  Once  the  contract  has  been  awarded  there  is  little 
concern  for  how  the  perfonnance  will  be  evaluated  or  what  happens  to  that  information. 
For  years  the  evaluation  factors  had  been  unclear  and  contractor  performance  reviews 
used  factors  that  were  vague,  and  the  reports  were  ultimately  not  appropriately  filed  due 
to  a  mismanaged  system  with  no  oversight  or  direction.  After  GAO  investigations,  the 
CPARS  and  PPIRS  systems  were  finally  put  on  track  and  government  agencies  have 
made  strides  in  proper  reporting  techniques.  Perfonnance  information  is  valuable  for 
future  acquisitions  for  the  same  reasons  as  other  non-cost  factors — the  reliability  of  the 
infonnation  provides  prized  data  that  can  mean  the  difference  for  reducing  risk  for  future 
contractor  poor  performance.  If  past  performance  information  is  weighted  as  more 
important  than  cost,  then  a  tradeoff  evaluation  would  rely  wholly  on  the  quality  of  that 
past  performance  information,  at  which  time  it  would  be  helpful  if  the  information 
captured  in  CPARS  was  not  only  present  but  also  reliable.  As  substantial  projects,  such  as 
the  SSIP  test  program,  depend  more  heavily  on  accurate  and  reliable  CPARS  reporting, 
there  is  likelihood  that  this  data  could  become  more  trustworthy  over  time. 

The  confirmed  statistical  relationships  between  the  choice  of  source  selection 
methodology  and  contract  outcomes  was  inconclusive  given  the  design  of  this 
experiment.  More  data  is  needed  to  confirm  beyond  refute  that  there  is  a  relationship 
between  the  choice  of  utilizing  either  LPTA  or  tradeoff  and  the  quality  of  the  contract 
outcomes.  The  discovery  of  relationships  between  other  factors  (contract  value  and 
individual  CPARS  ratings)  was  an  unexpected  benefit  from  this  study.  There  appears  to 
be  strong  relationships  between  PALT  and  contract  value  for  varying  degrees  across  both 
contract  methodologies,  as  previously  discussed.  Further  research  is  necessary  to  evaluate 
the  influence  of  these  factors  by  or  on  the  choice  of  methodology,  if  any. 

Management  in  the  world  of  contracting  never  seems  to  have  enough  time  to 

perform  what  needs  to  be  done  to  maintain  fair,  ethical  and  responsible  contracting.  One 

recommendation  from  this  study  is  to,  where  manageable,  factor  in  more  time  during  pre- 
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award  to  allow  for  more  use  of  tradeoff  methods.  LPTA  is  a  means  of  awarding  contracts 
faster,  with  little  negotiating  on  non-cost  factors,  but  without  this  approach  there  is  no 
telling  what  innovations  are  being  left  on  the  table.  Management  can  also  benefit  from 
this  study  insomuch  that  there  is  a  necessity  to  document  the  pre-award  and  post-award 
activities  related  to  contract  development  and  performance.  In  order  for  academia  to 
detennine  if  there  is  a  relationship  between  actions  and  outcomes  in  the  world  of 
contracting,  there  must  be  enough  real-world  data  that  can  be  analyzed.  At  this  time,  for 
the  variables  evaluated  in  this  study,  data  was  not  available  to  confirm  without  a  doubt 
that  a  pattern  exists.  Management  should  examine  where  gaps  in  the  data  exist  and  try  to 
bridge  those  gaps  to  ensure  benefit  from  future  research. 

B.  CONCLUSIONS 

The  results  of  this  study  indicate  that  the  only  area  where  the  choice  of  source 
selection  methodology  (LPTA  or  tradeoff)  produced  significantly  different  contracting 
outcomes  was  in  the  case  of  the  contractor’s  CPARS  cost  rating,  where  the  cost  rating 
was  worse  for  tradeoff  contracts.  More  data  and  a  larger  sample  is  needed  to  confirm  this 
notion. 

There  appears  to  be  a  strong  positive  relationship  between  tradeoff  acquisitions 
and  PALT  and  contract  value  and  PALT.  Length  of  contract  preparation  is  clearly 
impacted  to  a  degree  by  factors  other  than  just  the  choice  of  source  selection 
methodology.  Contract  complexity  and  dollar  value  are  likely  to  have  an  impact  on  the 
length  of  time  required  to  put  a  contract  into  place. 

There  appears  to  be  a  strong  negative  relationship  between  the  CPARS  ratings 
and  PALT,  suggesting  that  the  CPARS  ratings  decline  as  the  PALT  (number  of  days) 
increases.  This  may  be  the  result  of  an  increased  chance  of  contractor  negative 
perfonnance  the  longer  it  takes  for  a  contract  to  be  put  into  place,  possibly  due  to  a 
breakdown  in  communication  or  complexity  of  the  contract. 

There  is  only  a  moderately  positive  linear  relationship  between  PALT  and 
CPARS  ratings  for  LPTA  acquisitions,  indicating  that  for  acquisitions  with  a  longer 
PALT  time,  the  CPARS  rating  may  be  better.  This  may  be  the  result  of  better  efforts 
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being  applied  to  the  contract  setup  leading  to  better  contractor  performance  overall. 
There  is  also  a  negative  linear  relationship  between  the  contract  value  and  the  CPARS 
ratings  for  the  LPTA  acquisitions,  suggesting  that  as  the  dollar  value  of  the  contract 
increases,  there  is  higher  risk  for  more  criticism  of  the  deliverables,  therefore  might  be  a 
better  fit  for  tradeoff  methodology  over  LPTA.  There  is  a  strong  positive  linear 
relationship  between  PALT  and  contract  value  for  LPTA  acquisitions,  which  is  possible 
related  to  the  complexity  of  the  contract — the  higher  the  dollar  value  on  the  contract,  the 
more  complex  the  contract,  the  longer  the  PALT. 

For  tradeoff  acquisition  there  appears  to  be  a  negative  relationship  between  the 
PALT  and  the  CPARS  ratings  for  tradeoff  acquisitions,  indicating  that  as  the  PALT 
increases  the  CPARS  ratings  for  quality  and  business  relationships  are  negative.  This 
may  indicate  that  even  with  excessive  amounts  of  time  dedicated  to  contract  preparation 
(either  due  to  diligent  groundwork  or  redundant  reviews)  the  CPARS  ratings  are  not 
necessarily  better.  More  time  does  not  guarantee  better  reviews  for  the  increase  in  time 
dedicated  alone.  Further  analysis  of  the  CPARS  records  is  necessary,  as  well  as 
management’s  oversight  to  ensure  CPARS  records  are  being  created  in  accordance  with 
policy.  Finally,  further  research  is  needed  to  determine  if  the  small  sample  size  for  this 
experiment  is  skewing  the  results  and  statistical  relationships. 

C.  RESEARCH  QUESTION  ANSWERS 

In  Chapter  I  of  this  paper,  the  following  research  questions  were  posited: 

1 .  Does  the  utilization  of  LPTA  or  tradeoff  have  an  impact  on  the  outcome  of 
competitively  awarded  contracts? 

2.  Are  acquisition  teams  making  knowledgeable  decisions  regarding  which 
best  value  methodology  is  most  appropriate  for  their  source  selection? 

Question  #1  was  answered,  to  limited  extent,  with  the  sample  size  obtained. 
Based  on  the  results  of  the  statistical  analysis  with  this  small  sample  size,  there  is  no  firm, 
irrefutable  relationship  between  the  methodology  choice  of  LPTA  or  Tradeoff  and  the 
contract  outcomes.  This  study  should  be  performed  again  with  a  larger  sample  size  to 
study  whether  a  relationship  does,  in  fact,  exist.  The  small  sample  size  may  have 
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precluded  a  MANCOVA  and  a  proper  ANCOVA,  but  the  correlation  tables  (Table  3  in 
Chapter  IV)  point  to  relationships  that  are  worth  investigating. 

Question  #2  cannot  be  answered,  given  the  lack  of  a  clear  relationship  between 
the  methodology  and  contract  outcomes.  There  is  no  conclusive  indicator  for  whether  the 
acquisition  teams  are  selecting  the  appropriate  source  selection  methodology  based  on 
this  limited  data.  Given  that  the  sample  had  many  more  LPT  A  than  tradeoff  source 
selections,  it  seems  that  PEO  EIS  is  selecting  LPTA  methods  more  frequently.  Given  also 
that  the  sample  was  comprised  of  larger  dollar  IT  acquisitions,  one  would  assume  that 
tradeoff  might  be  a  better  option  for  achieving  best  value  for  the  government.  This 
assumption  also  precludes  a  necessity  for  innovative  solutions;  if  the  requirement  is 
relatively  standard  for  this  organization  there  may  not  be  a  desire  for  innovation,  nor  a 
willingness  or  necessity  to  consider  non-price  factors.  LPTA  is  an  appropriate  choice  for 
COTS  purchases,  which  comprised  the  majority  of  the  sample,  therefore  indicating  that 
LPTA  was  appropriately  used  for  this  sample  of  contracts.  Further  review  of  pre-award 
decision-making  documentation  of  a  larger  sample  size  could  shed  more  light  on  the 
reasons  the  acquisition  team  chose  LPTA  more  frequently  than  trade  off.  The  sample  of 
19  contract  actions  was  comprised  of  14  LPTA  actions  and  only  five  tradeoff  actions, 
indicating  that  LPTA  is  being  utilized  more  frequently  than  tradeoff.  A  larger,  more 
diverse  sample  is  necessary  to  get  a  better  picture  for  how  methodologies  are  selected  for 
a  wider  range  of  acquisitions. 

D.  MANAGERIAL  IMPLICATIONS  AND  RECOMMENDATIONS 

The  choice  of  source  selection  methodology  does  appear  to  have  some  impact  on 
contract  outcomes.  Managers  should  stress  the  importance  of  this  decision  with 
acquisition  teams  entering  the  source  selection  phase  of  the  procurement,  urging  the  team 
to  consider  the  choices  carefully.  The  choice  of  course  selection  strategy  should  befit  the 
requirement.  More  data  is  needed  to  pinpoint  the  precise  effects  of  strategy  on  contract 
outcomes.  While  some  DOD  organizations  specialize  in  purchasing  bullets,  tanks,  or 
other  tangible  goods,  all  organizations  at  some  point  purchase  IT,  whether  in  the  form  of 
simple  laptop  purchases  or  more  complex  services.  All  organizations  therefore  should 
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have  an  interest  in  learning  how  to  purchase  IT  in  ways  that  gamer  the  best  value  for  the 
government.  This  study  evaluated  a  single  organization  with  an  IT -purchasing  profile 
(numerous  COTS  purchases),  resulting  in  findings  that  can  be  applied  to  other 
organizations  making  similar  purchases. 

On-the-job  training  that  supports  diverse  contracting  experiences  is  crucial  for 
developing  acquisition  professionals.  LPTA  was  found  to  be  utilized  more  often  than 
tradeoff  in  this  study  as  there  is  further  utilization  of  orders  against  indefinite  delivery 
indefinite  quantity  (IDIQ)  contracts,  General  Services  Administration  (GSA)  contract 
vehicles,  and  Blanket  Purchase  Agreements  (BPAs),  which  utilize  streamlined 
contracting  in  the  form  of  orders  to  award  requirements  faster.  Under  IDIQs,  GSA 
vehicles  and  BPA  agreements  the  rates  are  typically  pre-negotiated  with  the  set  of 
vendors,  leaving  little  necessity  for  negotiation  and  tradeoff  tactics.  While  tradeoff  can  be 
utilized  as  a  source  selection  methodology,  there  could  be  a  tendency  to  use  LPTA 
instead  of  tradeoff  to  facilitate  an  easier  evaluation.  Increased  use  of  these  simplified 
order  vehicles  may  be  the  result  of  inexperienced  contracting  professionals  relying  on 
pre-established  vehicles  to  streamline  acquisitions.  There  is  no  problem  with  utilizing 
established  vehicles,  other  than  the  hindrance  of  experiencing  successful  tradeoff 
acquisitions.  Managers  should  stress  the  importance  of  on-the-job  training  with  diverse 
experiences  to  help  personnel  gain  experience  and  confidence  to  explore  non-simplified 
acquisitions  where  tradeoff  tactics  can  be  implemented  successfully. 

The  documentation  for  simple  COTS  purchases  is  often  abbreviated,  since  the 
collective  files  are  likely  stored  together  by  the  contracting  officer,  thereby  creating  a 
larger  picture  of  the  entire  contract.  As  contracting  officers  change,  or  decentralized 
contracts  are  utilized  by  multiple  contracting  officers  and  organizations,  the 
documentation  can  become  murkier  and  less  clear  for  someone  reviewing  the  files 
without  any  context.  The  future  of  contract  analysis  relies  on  clear  documentation  and 
accessibility  of  contract  files  for  review.  A  great  challenge  in  this  study  was  finding  not 
only  a  sample  that  could  accommodate  the  statistical  equations,  but  merely  finding 
complete  contract  files  and  CPARS  reports.  PEO  EIS  has  experienced  multiple 
contracting  personnel  support  changes  over  time,  which  results  in  incomplete  contract 
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files  and  incomplete  CPARS  reports.  Management  in  other  organizations  can  benefit 
from  this  study  by  creating  clear  regulations  and  internal  guidance  for  storage  of  contract 
file  documentation,  particularly  as  contract  change  hands  change  over  time.  Proper  file 
maintenance  may  seem  an  administrative  concern,  but  as  systems  become  more  advanced 
and  are  able  to  process  trends  and  statistical  findings  from  file  data,  the  potential  for 
future  findings  is  limitless. 

E.  AREAS  FOR  FURTHER  RESEARCH 

A  constant  challenge  in  this  study  was  finding  enough  data  to  run  proper 
statistical  analyses.  The  initial  goal  was  to  have  at  least  14  cases  of  each  source  selection 
methodology.  Only  19  cases  were  acquired.  This  was  the  result  of  incomplete  or 
inaccessible  CPARS  reports,  frequent  use  of  sole  source/limited  source  procurements,  or 
inaccessible  contract  files.  While  useful  knowledge  was  obtained  from  the  scaled-down 
statistical  analyses,  there  may  be  other  patterns  that  would  emerge  from  a  larger  sample 
size. 

While  PEO  EIS  is  a  large  IT  organization  with  a  large  mission  in  the  Army,  PEO 
EIS  is  only  one  organization  in  the  DOD.  There  are  many  benefits  associated  with 
replicating  this  research  in  other  agencies  with  similar  and  with  different  purchasing 
organizations.  The  sample  for  this  research  contained  mostly  COTS  hardware  purchases. 
A  sample  of  service  contracts  that  tend  to  utilize  tradeoff  strategies  more  heavily  may 
show  very  different  results.  More  analysis  is  necessary  to  find  trends  that  span  the  entire 
purchasing  profile  of  the  DOD. 

Contract  documentation  in  both  pre-award  and  post-award  is  an  administrative 
function  that  necessitates  detail,  objectivity  and  dedication.  The  contract  manager  (either 
contracting  officer  or  COR)  is  responsible  for  ensuring  that  the  contract  file  contains  all 
necessary  data  for  review  and  analysis.  The  challenge  for  this  research  was  not  only 
gaining  access  to  the  contract  files  and  CPARS  records  (made  even  more  challenging  by 
the  changing  of  hands  between  contracting  agencies  over  the  past  five  years  of  PEO  EIS’ 
contracting  history),  but  ensuring  that  the  pieces  of  the  files  necessary  for  data  extraction 
were  complete  and  present.  Incomplete  CPARS  reports,  incomplete  contract  files,  and 
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inaccessible  contract  files  and  CPARS  records  were  major  hurdles  experienced  during 
data  collection. 

This  study  examined  a  limited  number  of  contract  outcomes  and  only  one 
covariate.  There  may  be  other  means  of  measuring  contract  outcomes,  such  as  earned 
value  management  (EVM).  As  stated  earlier,  this  effort  can  only  be  fulfilled  if  the 
contract  documentation  is  detailed  enough  to  support  a  sample  that  provides  all  such  data. 

While  the  goal  of  this  study  may  not  have  been  entirely  achieved,  headway  was 
made  in  the  field  of  best  value  contracting.  Acquisition  teams  need  to  be  prepared  to 
handle  the  unforeseen  challenges  of  the  acquisition  environment  and  be  prepared  with 
skills  to  procure  using  not  only  LPTA  but  also  tradeoff  strategies,  and  be  knowledgeable 
of  the  implications  of  choosing  one  method  over  another.  In  order  to  achieve  value,  the 
playing  field  must  be  set  up  to  accommodate  that  achievement;  the  DOD  needs  to  be 
training  acquisition  teams  to  handle  varying  degrees  of  methodologies  and  exercise 
strong  business  judgment  to  gain  the  benefits  of  innovation  where  possible.  To  harken 
back  to  Warren  Buffett’s  words,  “Price  is  what  you  pay;  value  is  what  you  get”  (Buffett, 
2008,  p.  5).  In  order  for  the  DOD  to  get  the  value  it  is  due,  acquisition  teams  need  to  be 
equipped  with  the  best  knowledge  regarding  implications  of  their  choices.  This  study  has 
begun  to  explore  the  impact  of  methodology  on  contractual  outcomes  and  has  only 
scratched  the  surface  of  what  the  world  of  best  value  contracting  has  to  offer. 
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